Disease diagnosis method and system based on multi-mode space-frequency domain adaptive fusion

Abstract

The invention discloses a disease diagnosis method and system based on multi-modal space-frequency domain adaptive fusion, and relates to the field of artificial intelligence and biomedical engineering.The method comprises the steps that multi-modal data are standardized, the unified and standardized multi-modal data are coded, and multi-modal initial feature representation is obtained; after projection and gating alignment and cross-modal interactive attention alignment are carried out on the initial feature representation of each modal, enhanced representations of each modal are obtained, and then the enhanced representations of each modal are fused into a shared feature representation; performing deep feature extraction on the enhanced representation of each mode to obtain deep features of each mode, and performing adaptive multi-domain feature enhancement processing to obtain multi-domain enhanced features of each mode; performing semantic alignment on the multi-domain enhanced features of each mode, and then performing fusion through a hierarchical attention mechanism to obtain fusion features; and the fusion features are input into a diagnosis network for prediction, a disease diagnosis result is obtained, and the intelligent diagnosis precision and robustness of papillary thyroid carcinoma are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and biomedical engineering, and in particular to a disease diagnosis method and system based on multimodal spatial-frequency domain adaptive fusion. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Papillary thyroid carcinoma (PTC) is the most common malignant thyroid tumor in clinical practice. Early auxiliary diagnosis typically relies on ultrasound imaging, pathological sections, laboratory tests, and relevant clinical text information. However, with the development of precision medicine, single-modal data is insufficient to comprehensively characterize the spatial structure, cellular state, and molecular features of tumors. Multimodal medical data fusion is gradually becoming a key pathway to improve the accuracy and stability of auxiliary diagnosis. Especially with the increasing use of multi-source data such as single-cell transcriptomics, hyperspectral imaging, medical text, and ultrasound video in clinical research, the construction of large-scale multimodal medical models has become a research hotspot. However, existing technologies still face several challenges that affect the effectiveness of intelligent auxiliary diagnosis of PTC: Significant heterogeneity exists among multimodal medical data. Single-cell transcriptome data reflects cellular heterogeneity at the gene expression level; hyperspectral images contain hundreds of spectral channels, characterizing the optical absorption and scattering properties of tissues; medical text data records doctors' diagnostic logic in an unstructured manner; and ultrasound videos contain dynamic changes in lesions as the probe moves. These four types of data differ greatly in dimensionality, data distribution, noise type, and structural representation, making their unified alignment in spatial, semantic, and statistical properties extremely difficult.

[0004] Cross-modal feature extraction is highly complex. Single-cell data emphasizes gene expression patterns, hyperspectral data emphasizes spectral curve morphology, text data emphasizes entity-medical relationships, and video data emphasizes spatiotemporal evolution patterns. Each modality focuses on different key information, making it difficult to balance independence and complementarity in feature extraction. Current methods often optimize for a single modality, lacking a universal model framework capable of uniformly handling feature structures across different modalities.

[0005] Multimodal feature fusion lacks robustness and adaptability. Traditional fusion strategies often employ simple splicing, weighting, or shallow interaction methods, failing to adequately handle the distributional differences between modalities. This can easily lead to fusion results biased towards modalities with larger data volumes or lower noise levels. Especially in disease diagnosis, the importance of different modalities varies with patient characteristics and lesion presentation. Fixed-weight fusion methods struggle to achieve dynamic adaptation, resulting in insufficient stability of auxiliary diagnostic results.

[0006] Self-attention mechanisms face adaptation challenges in multimodal semantic interactions. While self-attention structures can capture long-range dependencies, directly inputting multimodal features into standard multi-head attention can lead to an imbalance in feature coupling. For example, single-cell high-dimensional gene features and video temporal features differ significantly in scale and meaning; self-attention may incorrectly assign invalid cross-modal correlations, resulting in model instability or a surge in computational complexity.

[0007] Multimodal diagnostic results lack hierarchical consistency constraints. PTC diagnosis typically involves the cellular level (e.g., cell subtype expression characteristics), tissue level (e.g., spectral reflectance properties), imaging level (e.g., dynamic ultrasound manifestations), and textual level (e.g., physician-described semantics). Existing methods often process information from one or a few levels, lacking hierarchical modeling of the relationship between molecules, tissues, images, and semantics. This results in poor interpretability of diagnostic results, failing to meet clinical needs. Physicians cannot make decisions based on isolated, uninterpreted results, leading to low utilization of diagnostic outcomes.

[0008] In summary, existing multimodal medical diagnostic methods suffer from insufficient modality alignment capabilities, making it difficult to establish connections between genes, spectra, semantics, and spatiotemporal images; the fusion strategies lack hierarchy and dynamism, resulting in models being unable to adapt to the modal differences among different patients; the interaction capabilities between high-dimensional modalities (such as single-cell and HSI) and temporal modalities (video) are insufficient, leading to low information utilization; the model structures are mostly based on traditional CNNs or simple Transformers, lacking cross-modal unified representation capabilities; and the lack of cross-modal consistency constraint mechanisms affects diagnostic stability and clinical interpretability, failing to provide doctors with effective decision-making basis. Summary of the Invention

[0009] To overcome the shortcomings of the prior art, this invention provides a disease diagnosis method and system based on multimodal spatial-frequency domain adaptive fusion, aiming to improve the accuracy and robustness of intelligent diagnosis of papillary thyroid carcinoma and effectively assist doctors in decision-making.

[0010] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, the present invention provides a disease diagnosis method based on multimodal spatial-frequency domain adaptive fusion, comprising: Acquire multimodal data and perform standardization processing on each data point to obtain unified and standardized multimodal data; The unified and standardized multimodal data are encoded separately to obtain the initial feature representation of the multimodality. After projection and gating alignment and cross-modal interactive attention alignment of the initial feature representation of each modality, the enhanced representation of each modality is obtained. Then, the enhanced representations of each modality are fused into a shared feature representation. Deep feature extraction is performed on the enhanced representation of each modality to obtain the deep features of each modality. After adaptive multi-domain feature enhancement processing, the multi-domain enhanced features of each modality are obtained. After semantic alignment of the multi-domain enhancement features of each modality, they are fused through a hierarchical attention mechanism to obtain fused features; The fused features are input into the diagnostic network for prediction, resulting in a disease diagnosis.

[0011] In a further technical solution, the multimodal data includes single-cell transcriptome data, hyperspectral images, text data, and video data.

[0012] Further technical solutions to obtain unified and standardized multimodal data are as follows: A cell adjacency graph was constructed based on single-cell transcriptome data. After clustering the gene sets, several gene blocks were obtained. Each gene block was smoothed and normalized on the cell adjacency graph. The smoothed gene blocks were corrected based on the local bias correction factor to obtain the standardized expression of each gene block. After merging, a single-cell expression matrix was obtained. After the hyperspectral image is divided into spatial blocks, multiple spatial blocks are obtained. The channels within each spatial block are normalized based on the local statistics and block offset of each spatial block. The normalized spatial blocks are corrected based on the local statistics and spectral features of the reconstruction residual to obtain the corrected spatial blocks. After merging, a normalized hyperspectral cube is obtained. Entities are identified from text data and their corresponding concepts are obtained. A medical semantic graph is constructed and then input into a graph neural network for encoding to obtain a structured text representation. Frame-level feature extraction is performed on video data to obtain frame feature vectors. Candidate keyframes are selected from the frame feature vectors based on the comprehensive significance score. Redundancy is removed from the corrected candidate keyframes to obtain the final keyframes. The final keyframes are then encoded into frame-level feature sequences.

[0013] A further technical solution involves projecting and gating the initial feature representations of each modality, followed by cross-modal interactive attention alignment, to obtain the enhanced representations of each modality. Specifically: The initial feature representations of each modality are mapped to a unified dimension to obtain the feature representations of each modality; the feature representations of each modality are weighted based on the modality gating vector to obtain the weighted feature representations. Cross-modal attention is computed on the weighted feature representation to obtain the intermodal interaction representation; based on the intermodal interaction representation and the weighted feature representation, the enhanced representation of each modality is obtained.

[0014] A further technical solution employs composite loss:

[0015] in, For compound loss, , , , For hyperparameters, For modal reconstruction loss, For cross-modal contrast loss, To align the regularization loss, For parameter regularization terms.

[0016] A further technical solution, after adaptive multi-domain feature enhancement processing, obtains multi-domain enhanced features for each modality as follows: define a set of domain features and set domain attention weights; perform domain enhancement on the deep features of each modality based on the domain attention weights to obtain multi-domain enhanced features for each modality.

[0017] Further technical solutions yield the following specific fusion features: The multi-domain enhancement features of each modality are projected onto a unified embedding space to obtain a unified embedding vector for each modality. The unified embedding vectors of each modality are merged to obtain the unified embedding feature; Multi-layer self-attention operation is performed on the unified embedding features to obtain fused features.

[0018] Secondly, the present invention provides a disease diagnosis system based on multimodal spatial-frequency domain adaptive fusion, comprising: The standardization module is configured to acquire multimodal data and perform standardization processing on each data to obtain uniformly standardized multimodal data. The feature alignment module is configured to: encode the unified and standardized multimodal data separately to obtain the initial feature representations of the multimodal data; perform projection and gating alignment and cross-modal interactive attention alignment on the initial feature representations of each modality to obtain the enhanced representations of each modality; and then fuse the enhanced representations of each modality into a shared feature representation. The feature enhancement module is configured to: extract deep features from the enhanced representation of each modality to obtain deep features of each modality, and then obtain multi-domain enhanced features of each modality after adaptive multi-domain feature enhancement processing; The feature fusion module is configured to: semantically align the multi-domain enhanced features of each modality and then fuse them through a hierarchical attention mechanism to obtain fused features; The identification and diagnosis module is configured to input fused features into the diagnosis network for prediction to obtain disease diagnosis results.

[0019] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the disease diagnosis method based on multimodal spatial-frequency domain adaptive fusion as described in the first aspect.

[0020] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the disease diagnosis method based on multimodal spatial-frequency domain adaptive fusion as described in the first aspect.

[0021] The above one or more technical solutions have the following beneficial effects: This invention effectively solves key challenges in multimodal biomedical data fusion, such as heterogeneity, complementarity mining, and efficient utilization, through a series of innovative data processing, feature alignment, multi-domain enhancement, and hierarchical fusion technologies. It significantly improves the accuracy, robustness, and interpretability of cell identification and disease diagnosis, providing doctors with understandable and trustworthy auxiliary decision-making results, enhancing diagnostic confidence, and reducing decision uncertainty.

[0022] For multimodal data in cell identification, this invention innovatively designs a standardized processing flow for each modality, improving data quality and comparability at the fundamental data level. For single-cell transcriptome data, gene block graph smoothing and local bias correction suppress technical noise while precisely preserving key biological variations characterizing cell subpopulations, avoiding information loss caused by over-smoothing. For hyperspectral images, block-based adaptive normalization and spectral residual correction effectively overcome local illumination unevenness and scattering interference, restoring true spectral features and significantly improving the reliability of image information. For medical text data, a medical ontology is introduced to construct a semantic graph, transforming unstructured text into a structured representation rich in medical prior knowledge, resolving terminological ambiguity and expression diversity issues, and making it easy to align with image and gene data. For video data, saliency-driven and temporal consistency normalization intelligently selects diagnostic keyframes and corrects time drift, preserving the most diagnostically valuable dynamic information while compressing data volume.

[0023] This invention employs a multi-layered, progressive alignment strategy, including projection and gating alignment, cross-modal interactive attention alignment, and semantic alignment, to force different modal information to align and interact in the semantic space during the early and middle stages of feature extraction. This method can more effectively capture complex cross-modal correlations and consistency information, providing a high-quality, highly consistent feature foundation for subsequent fusion and recognition, fundamentally improving the depth and accuracy of fusion. Furthermore, a composite loss function combining modality reconstruction loss, cross-modal contrast loss, and alignment regularization loss is designed for joint optimization, enabling the model to simultaneously consider the effective reconstruction of information from each modality, cross-modal semantic convergence and differentiation, and feature space regularization during training.

[0024] This invention innovatively introduces adaptive multi-domain feature enhancement processing, enabling the model to adaptively focus on the combination of feature dimensions most critical to the current recognition task, thereby enhancing the feature representation capability.

[0025] In the final fusion stage, this invention employs a hierarchical attention mechanism with semantic alignment to improve the flexibility and effectiveness of the fusion process, and also endows the model with a certain degree of interpretability. Attached Figure Description

[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0027] Figure 1 This is a flowchart of a disease diagnosis method based on multimodal spatial-frequency domain adaptive fusion according to an embodiment of the present invention. Detailed Implementation

[0028] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0029] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0030] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0031] Current multimodal medical diagnostic technologies have significant limitations in modality alignment, feature mining, cross-modal interaction, and decision interpretability. There is an urgent need to construct a large medical model structure that can deeply integrate single-cell transcriptome, hyperspectral images, medical text, and video data to improve the accuracy and robustness of intelligent diagnosis of papillary thyroid carcinoma.

[0032] Example 1 like Figure 1 As shown, this embodiment discloses a disease diagnosis method based on multimodal spatial-frequency domain adaptive fusion. It uses a large medical model for disease diagnosis to assist doctors in making treatment decisions. The large medical model includes five modules, as detailed below: Multimodal normalization preprocessing module: performs heterogeneity elimination and normalization processing on four types of data: single-cell transcriptome, hyperspectral images, medical text, and video. It adopts AGBN, BCAR-SURC, OAMN, and SKTCN methods, and combines cross-modal contrast calibration (CMCC) to improve consistency.

[0033] Cross-modal feature encoding and alignment module: Features are extracted using dedicated encoders (GNN, dual-path convolution, medical Transformer, spatiotemporal Transformer), and feature alignment and fusion are achieved through unified projection, modal gating, and cross-modal interactive attention.

[0034] Adaptive multi-domain feature enhancement module: Extracts deep features through GCN, spatial / spectral convolution, 3D-CNN, etc., and introduces domain attention mechanism to dynamically enhance key feature domains.

[0035] Semantic alignment and hierarchical attention fusion module: Projects multi-domain features onto a unified semantic space, achieves deep fusion through multi-layer self-attention, and maintains structural consistency.

[0036] Diagnostics and Interpretability Module: Uses fully connected networks for classification prediction and provides interpretation of each modality's contribution through gradient analysis.

[0037] The method includes the following steps: S1: Acquire multimodal data and perform standardization processing on each data point to obtain unified standardized multimodal data; In this embodiment, multimodal data is acquired from multiple sources and standardized to ensure comparability, alignment, and preservation of weak diagnostic signals. The multimodal data includes single-cell transcriptomes (scRNA-seq), hyperspectral images (HSI), medical text, and ultrasound / narrowband video.

[0038] To overcome intermodal heterogeneity, this invention proposes a hybrid scheme of pixel / unit-level adaptive mapping, block statistical reconstruction, ontology-enhanced text standardization, and temporal saliency normalization. It also introduces cross-modal self-supervised contrast calibration as a global consistency constraint, so that the output multimodal representation retains its own characteristics and is fusionable.

[0039] The specific steps are as follows: S101: Collect multimodal data and register metadata, including device ID, collection parameters, timestamp, batch information, etc.

[0040] S102: Perform preliminary cleaning on the data of each modality, including denoising, removal of bad frames / low-quality cells, and annotation of missing values, to obtain preliminary data for each modality.

[0041] S103: Preprocess and standardize the preliminary data of each modality to obtain standardized multimodal data.

[0042] (1) Single-cell transcriptome data (scRNA-seq) This paper proposes an innovative method called Adaptive Graph-Block Normalization (AGBN) based on local graph smoothing. This method constructs cells as a graph structure and performs adaptive gene-block normalization on the graph. By combining local graph smoothing with gene block statistical correction, single-cell transcriptome data are normalized. This approach suppresses sequencing noise while preserving subpopulation diversity.

[0043] 1) Based on single-cell transcriptome data, a cell adjacency graph G=(V,E) is constructed. By calculating the distance between different cells in the gene expression vector space, cells with similar expression patterns are assigned larger connection weights, while cells with large expression differences are assigned smaller connection weights, thereby characterizing the biological similarity relationship between cells in the cell adjacency graph.

[0044] In a cell adjacency graph, nodes are cells, and edge weights are determined based on expression similarity, expressed as follows:

[0045] in, For cells and cells Weights (edges) between them. For cells The original expression level, For cells The original expression level, This is the Gaussian kernel width parameter.

[0046] 2) Gene-blocks are obtained by performing spectral clustering on gene sets in single-cell transcriptome data. .

[0047] 3) For each gene block Graph Laplacian smoothing normalization is performed on the cell adjacency graph to obtain smoothed gene blocks. , represented as:

[0048] in, Optimize variables for the smoothed representation matrix. This is the expression matrix for the corresponding gene block. For the smoothing intensity (which can be learned or determined through cross-validation), L=DW is a graph Laplace. The analytical solution to this equation is:

[0049] in, It is an identity matrix.

[0050] 4) After smoothing, perform adaptive standardization on each gene block and define a local bias correction factor:

[0051] in, For the first The gene block in the first Local central tendency value in individual cells For median operations, For the first The gene block in the first Expression vectors in individual cells For the first The gene block in the first The degree of local dispersion in individual cells This represents the absolute deviation of the median.

[0052] Normalized expression of each gene block was calculated based on the local bias correction factor:

[0053] in, For the first The gene block in the first Standardized gene expression vectors in individual cells To prevent numerical stability terms from being divided by zero, all gene blocks are finally merged to obtain a single-cell expression matrix. .

[0054] (2) Hyperspectral image (HSI) The hyperspectral image is normalized by blockwise channel adaptive reconstruction with spectral unmixing residual correction (BCAR-SURC) to obtain a normalized hyperspectral image.

[0055] HSI contains multiple bands but there are inter-band correlations and local illumination / scattering differences. Traditional channel-by-channel normalization or global PCA denoising will lose local spectral features or cause boundary artifacts. This invention proposes BCAR-SURC, which decomposes the hyperspectral cube into spatial blocks and spectral segments. First, local channel block normalization is performed, and then the spectral non-negative mixture model (NMF) residual correction is performed to restore the true spectral composition.

[0056] 1) Hyperspectral images The block set is obtained by spatial partitioning (size p×p) and band segmentation (each segment contains several adjacent bands). .

[0057] 2) Calculate local statistics (maximum, minimum, variance) for each block and introduce a block offset term. , represented as:

[0058] in, , For block adaptive learning parameters, For the first The maximum value of all feature values ​​within each feature block For the first The minimum value of all feature values ​​within a feature block. For the first The variance of the feature distribution within each feature block.

[0059] 3) Perform intra-block channel normalization based on local statistics and block offset terms:

[0060] in, This represents the normalized eigenvalues ​​of the channels within the block. These are the original eigenvalues. , For spatial location indexes in the feature map, For feature channel index, It is a preset positive real constant.

[0061] 4) Perform spectral nonnegative mixture decomposition (NMF) on each spatial block to estimate the fundamental spectral matrix. With abundance matrix :

[0062] in, For the first The block-level normalized hyperspectral feature matrix of each spatial block.

[0063] Calculate the reconstructed residual :

[0064] Reconstructing residuals Used to characterize the The hyperspectral information in each spatial block that is not fully represented by the fundamental spectrum matrix and the abundance matrix contains local anomalous spectral responses and fine-grained difference features, which can serve as an important supplement for subsequent feature fusion and diagnostic discrimination.

[0065] 5) Correct the normalization results using the local statistics and spectral characteristics of the reconstructed residuals:

[0066] in, For the corrected space block, For adaptive attenuation coefficient, This is a residual filtering transform (e.g., wavelet or local Fourier low-pass).

[0067] 6) Merge all blocks to obtain a normalized hyperspectral cube. .

[0068] (3) Text data The text data is standardized by using Onto-Augmented Multi-granular Normalization (OAMN) enhanced by medical ontology to obtain standardized text data.

[0069] Clinical texts are often unstructured, diverse in expression, and contain significant entity synonyms and semantic ambiguities. Directly using general language models cannot fully utilize medical priors and clinical terminology systems, affecting semantic alignment with images / genes. This invention proposes OAMN, which introduces medical ontology (such as UMLS / ICD / BioPortal) mapping and multi-granularity entity aggregation during text cleaning and encoding, and maps text tokens to a structured medical semantic graph (entity-relation graph) to facilitate subsequent cross-modal alignment.

[0070] 1) Preprocess the text data, including sentence segmentation, word segmentation, stop word filtering, and spell correction, to obtain the original token sequence. .

[0071] 2) Utilizing medical NER models to analyze token sequences Identifying entities And find the corresponding concept ID in the medical ontology. .

[0072] 3) Aggregate the occurrences of the same concept ID in different sentences / paragraphs to form entity frequency and contextual statistical vectors. . Primarily used as an auxiliary statistical feature of text modalities for information summarization and interpretability.

[0073] 4) The medical semantic graph is constructed based on text entity and relation extraction and prior relation information from the knowledge base. It reflects both statistical co-occurrence and semantic associations within the medical field. (Constructing the medical semantic graph...) Nodes are entity concepts, edges are relations (e.g., located, accompanied, size, etc.), and weights are assigned to the edges:

[0074] in, For entities and The edge weights between elements represent the strength of the relationship. The type weight coefficient of the edge. For point mutual information, measurement and Statistical dependencies of co-occurrence in the corpus; Co-occurrence frequency, representing the frequency of occurrence in the text. and The number of times they occur simultaneously; , These are two entity concept nodes in the medical semantic graph.

[0075] 5) Encode the medical semantic graph into an embedding. :

[0076] in, For graph neural network encoding functions, This represents the number of layers in a graph neural network.

[0077] 6) Output standardized text data in a structured manner (structured text representation) ,in Standardized values ​​for numerical fields such as diagnostic labels and measurement values ​​(with unified units and anomaly handling).

[0078] (4) Video data Ultrasound videos contain numerous redundant frames, probe angle variations, and gain changes. Directly using all frames would incur computational burdens and be susceptible to transient noise interference. Furthermore, the brightness / contrast of the video drifts over time, affecting alignment with static modalities. This invention innovatively proposes Saliency-Driven Keyframe & Temporal Consistency Normalization (SKTCN), which selects keyframes based on multi-scale saliency and structural similarity, and performs global consistency normalization on temporal brightness / contrast drift to maintain semantic consistency between video and static images.

[0079] 1) Frame-level feature extraction. For each frame of the video... Edge map (Canny), local texture (LBP), and optical flow information are extracted and combined to form a frame feature vector. .

[0080] 2) Saliency Score. Calculate the overall saliency score of the feature vectors for each frame, expressed as:

[0081] in, To calculate the significance score, , , These are the weighting coefficients for edge, texture, and optical flow features, used to adjust the contribution ratio of each saliency component; and They represent the first Frame edge saliency features and texture saliency features, This represents the optical flow vector field of this frame. is the L1 norm of the optical flow vector, used to reflect the intensity of motion between frames.

[0082] Select those with a comprehensive significance score greater than the threshold. The frame feature vectors are used as candidate keyframes, or the frame feature vectors are sorted from largest to smallest based on their comprehensive significance score, and the top ones are selected. The frame feature vectors are used as candidate keyframes to form a candidate keyframe set. .

[0083] 3) Temporal Consistency Normalization. Calculate the temporal luminance drift curve b(t) for the video sequence (e.g., in the video sequence, calculate the local average of the luminance or pixel intensity for each frame, and then observe the trend of these averages changing with frame number (time), and fit a smoothing function). (such as low-order polynomials or splines), and then correct each candidate keyframe:

[0084] in, These are the corrected candidate keyframes. For the original keyframes, For frame local contrast estimation, This is a preset constant.

[0085] 4) Keyframe Refinement. The corrected candidate keyframes are further refined using Structural Similarity (SSIM) to remove redundancy, retaining the frame closest to the HSI / pathology ROI structure as the final keyframe. .

[0086] 5) Encode the final keyframes into frame-level feature sequences. ,in For convolutional or Transformer feature extractors, This is the final set of keyframes selected.

[0087] S104: Perform cross-modal self-supervised comparison and calibration (global consistency constraint) on the standardized multimodal data, and output unified standardized multimodal data, i.e., a unified standardized mode set. .

[0088] After completing the modality-specific standardization, cross-modal self-supervised contrastive calibration (CMCC) is applied to further bring the representations of the same anatomical location in different modalities closer together in the latent space.

[0089] For any modality pair (a,b) (e.g., HSI and keyframe, scRNA and lesion ROI), define the encoder output. and Corresponding to the same space / patient sample point Construct contrastive loss:

[0090] in, To compare the losses, and Samples In modality and modality Encoder input embedding below, Temperature parameter used to scale similarity; It is used for negative sample embedding to ensure the distinguishability of different samples. The contrast loss is jointly optimized with the reconstruction or smoothing loss within each modality to ensure alignment without destroying the information within the modality.

[0091] S105: Output unified and standardized multimodal data, i.e., multimodal sets. ,in, This is a single-cell expression matrix. It is a hyperspectral cube. For structured text representation. It is a frame-level feature sequence.

[0092] And provide the corresponding metadata (coordinate mapping, timestamp, batch label, normalization parameters { }), used for reversible or traceable alignment initialization of S2 (constructing a cross-modal unified coding structure).

[0093] S2: Encode the unified and standardized multimodal data separately to obtain the initial feature representation of the multimodality; after projecting and gating the initial feature representation of each modality and performing cross-modal interactive attention alignment, the enhanced representation of each modality is obtained, and then the enhanced representation of each modality is fused into a shared feature representation; In this embodiment, four types of modal data are unified and standardized. Each modality is encoded into a comparable latent representation of the same dimension, and semantic compatibility and discriminative ability are ensured through a designed alignment and fusion module. Structurally, this invention proposes a modular four-channel dedicated encoder, projection and gating alignment, and a cross-modal attention aligner (UME), along with clearly defined mathematical formulas and interpretable parameters. All parameters following the formulas are explained item by item at the end of the paper for easy review and reproduction.

[0094] S201: Based on the corresponding dedicated encoder, the unified and standardized multimodal data is encoded to obtain the initial feature representation of the multimodality.

[0095] The four types of standardized modal data are input into the corresponding dedicated encoders and encoded into initial feature representations:

[0096] in, This represents the initial characteristics of single-cell expression. This represents the initial characteristics of the hyperspectral spectrum. This represents the initial features of the text. This represents the initial feature representation of the video keyframes. Let be the initial feature dimensions output by each encoder. Cell count, For HSI pixels or patch count, For text tokens / entity counts, This refers to the number of keyframes. , , , Dedicated encoders for single-cell transcriptomics, hyperspectral images, text, and video, respectively. , , , These are the parameters for the corresponding dedicated encoder.

[0097] scEncoder: Based on a composite structure of graph neural networks (GNNs) and self-attention, it utilizes both local neighborhood information and captures global gene expression patterns on the cellular graph structure. A bidirectional cell-gene attention layer is applied after the GNN layer to couple and encode cellular nodes with subsets of key genes, thereby obtaining biologically interpretable cellular representations.

[0098] HSEncoder employs a dual spectral-spatial path. The spectral path uses 1D convolution and spectral attention, while the spatial path uses 2D pyramidal convolution. Finally, it obtains the spectral-spatial vector for each pixel / ROI through cross-scale fusion. A "band importance weight learner" is introduced into the spectral path to automatically suppress anomalous bands.

[0099] TextEncoder: Based on a medical pre-trained Transformer, with an external ontology embedding injection layer, it injects the semantic graph information generated by OAMN into the self-attention keys / values ​​of the Transformer, thereby improving the quality of medical semantic alignment.

[0100] VidEncoder: A spatiotemporal Transformer (alternating between temporal and spatial attention) with added saliency weights (from SKTCN) in the temporal dimension to amplify the impact of diagnostic frames.

[0101] S202: In the unified projection layer, the initial feature representations of each modality are linearly / nonlinearly mapped to a unified dimension to obtain the feature representations of each modality:

[0102] in, For the feature representation of each mode, The projection matrix is ​​learnable. For bias terms, .

[0103] S203: Modal Gating and Intensity Recalibration. To reflect the issue that "the contribution of different modalities varies in a given patient / sample," a modal gating vector is designed. :

[0104] And weight the feature representations for each modality:

[0105] in, For modal gating weights, It is the sigmoid activation function. For linear mapping weight vectors, To perform global averaging or attention pooling on the modal instance dimension (e.g., rows), For bias terms, This represents the weighted modal characteristics. The output of the gated control. ∈(0,1) can adaptively suppress low-quality or redundant modes.

[0106] S204: Cross-modal interaction attention alignment. The weighted modal feature representations are input into the cross-modal attention module to calculate the inter-modal interaction representations. Taking the interaction between two modalities a and b as an example:

[0107] in, and This is the weighted modal feature representation. , , It is a linear mapping matrix. , , For query, key, and value vectors.

[0108] The four modalities were paired for interaction, and the cross-modal enhanced representation of each modality was finally constructed (taking single-cell transcriptome data as an example, the cumulative result in the formula is the result of pairwise interaction between single-cell transcriptome data and the other three modalities):

[0109] in, This represents the single-cell transcriptome modality representation after cross-modal alignment.

[0110] Introducing structural constraint matrices into attention (Based on spatial / coordinate mapping from S1 and cell-pixel coarse registration information), the attention score is modified as follows:

[0111] in, To score attention, The structural constraint strength coefficient, Information derived from spatial / coordinate mapping and coarse cell-pixel registration is used to introduce spatial or structural priors into attention computation. Structural constraints can incorporate prior location information (such as the correspondence between lesion ROI and cell population) into attention, making the interaction not a completely free match but subject to biological / anatomical constraints.

[0112] S205: Fusion and Regularization of Shared Representations. The representations of each modality after interaction are then mapped to a unified shared representation:

[0113] in, For shared feature representation, This involves concatenation + MLP or multi-head attention integration. Shared feature representation. It is used to perform unified alignment and regularization of multimodal features during training to ensure consistency and information preservation of each modality in the latent space.

[0114] To ensure alignment and modal information preservation, a composite loss method is used during the training of the large medical model:

[0115] in, For compound loss, , , , These are hyperparameters that control the weights of the corresponding loss terms. For modal reconstruction loss, For cross-modal contrast loss, To align the regularization loss, The parameter regularization term is a regularization loss applied to the learnable parameters of the model. It is used to constrain the size or distribution of the parameters and prevent overfitting. It does not depend on the labels of the training data, but only on the model parameters themselves.

[0116] The learnable parameters of the entire medical model are jointly trained, and the composite loss simultaneously constrains modal information preservation, cross-modal consistency, shared space alignment and parameter regularization during the training process, thereby improving the robustness and diagnostic performance of multimodal features.

[0117] (1) Modal reconstruction loss (preserving internal modal information)

[0118] in, For the corresponding modality decoder, To unify and standardize multimodal data.

[0119] (2) Cross-modal contrastive loss (bringing paired samples closer together and distinguishing unpaired samples) 4 in, Different modal representations of the same patient / ROI This is the temperature coefficient.

[0120] (3) Alignment Regularization (MSE) (ensures that paired embeddings are close)

[0121] in, The weights are settable for modal pairs (which can be learned or set empirically).

[0122] (4) Parameter regularization term (to prevent overfitting) , , , These are hyperparameters that control the weights of the corresponding loss terms.

[0123] S206: Cell-Image Coarse Registration Module. To enable a coarse mapping between single-cell information and image pixels / patches, a learnable cross-domain attention mapping matrix is ​​designed. :

[0124] After mapping, an abundance map of cells to pixels can be constructed. Or the activation distribution of pixels to cells. The innovation of this module is to sparsify the attention based on biological spatial priors and use sparsity regularization. Controlling the sparsity of the mapping is beneficial for interpretability (some cell populations correspond only to specific ROIs).

[0125] S207: Output: Intermediate representation after modal alignment Unified sharing representation Modal gating coefficients Cell-pixel mapping matrix M and its sparse constraint parameters.

[0126] S3: Deep feature extraction is performed on the enhanced representation of each modality to obtain the deep features of each modality. After adaptive multi-domain feature enhancement processing, the multi-domain enhanced features of each modality are obtained. In this embodiment, after completing cross-modal unified encoding, deeper feature extraction is performed on the data of each modality, enabling it to capture valuable information for the diagnosis of papillary thyroid carcinoma in the spatial, frequency, spectral, and temporal domains. This invention proposes an Adaptive Multi-Domain Enhancement (AMDE) mechanism to dynamically enhance key domain features of different modalities, improving the internal representation capabilities of modalities and their subsequent fusion performance.

[0127] S301: Single-cell transcriptome characteristics Based on unified coding vector Graph Convolutional Networks (GCNs) were used to extract gene regulatory structural features:

[0128] in, To extract high-level semantic features of gene regulation, This is a cell adjacency diagram.

[0129] S302: Hyperspectral Image Features For hyperspectral unified vector Simultaneous spatial and spectral domain convolution extraction:

[0130] in, This is the final hyperspectral feature representation after fusing spatial and spectral domain features, used for subsequent cross-modal fusion and diagnostic analysis; The spatial texture of cell structure. To extract spectral absorption characteristics related to tissue lesions.

[0131] S303: Text Features For the unified coding vector The semantic extraction layer is used to further enhance the pathological description:

[0132] in, This is the final representation of the text modality. This is the semantic extraction layer.

[0133] S304: Video Features Spatiotemporal convolution (3D-CNN) is used to further obtain temporal and spatial motion patterns:

[0134] in, This is a spatiotemporal feature representation after fusion, used for subsequent cross-modal fusion and diagnostic analysis.

[0135] To address the issue of uneven contributions of various modal features across multiple domains (spatial / frequency / spectral / temporal domains), this invention proposes an adaptive multi-domain feature enhancement mechanism.

[0136] 1) Structural definition For any mode, assuming its features come from K domains (e.g., HSI comes from both spatial and spectral domains), then the domain feature set defined in this invention is as follows:

[0137] And through a learnable domain attention weight:

[0138] in, For the first Adaptive weights for each feature domain For the domain weight parameter vector, For the first Feature vectors of each feature domain For the first The eigenvectors of each feature domain.

[0139] 2) Domain Enhancement Output Final modal features Represented as:

[0140] Among them, the weight size It automatically adapts to the current data; feature enhancement is a dynamic, soft-selection mechanism. The model automatically strengthens "key domains" (such as the semantic domain of text and the spectral domain of HSI), thus achieving cross-modal adaptation and unified multi-domain enhancement. Ultimately, four types of modal enhancement features are obtained: , , , These will serve as inputs for the next step of multimodal semantic alignment and unified embedding space construction.

[0141] S4: Semantically align the multi-domain enhanced features of each modality and then fuse them through a hierarchical attention mechanism to obtain fused features; In this embodiment, semantic alignment of four modalities (single-cell transcriptomics, hyperspectral images, text, and video) is achieved, and fused using a hierarchical self-attention mechanism to obtain a unified high-dimensional feature representation that can ultimately be used for the diagnosis of papillary thyroid carcinoma. The entire process includes five sub-modules: (1) Cross-modal semantic alignment This enables semantic alignment of features across different modalities, ensuring that "the same pathological information is represented similarly across modalities." Feature enhancement is applied to each modality. Project onto a unified embedded space:

[0142] in, The projected uniform embedding vector. For modality-specific projection matrices, This is the bias vector. Alignment operations make features from different modalities semantically comparable, facilitating subsequent fusion.

[0143] (2) Construction of a multimodal unified embedding space Alignment features of four modalities , , , merge:

[0144] in, To unify feature representations in the embedding space and maintain the integrity of cross-modal information; These are modal weights, which control the contribution of each modality to the fused features.

[0145] (3) Layered self-attention fusion For unified embedding features Perform multi-layered self-attention operations:

[0146] in, For the first Layer fusion features For the first Multi-head self-attention module The number of attention layers. Through the self-attention mechanism, feature information from different modalities can interact and enhance each other, resulting in fused multimodal features.

[0147] (4) Maintaining cross-modal structural consistency The fusion features preserve the structural information of each modality through graph regularization constraints:

[0148] in, The graph regularization loss is used to maintain the structural consistency of the fused features; The Laplacian matrix of the feature adjacency graph ensures structural consistency of the fused features.

[0149] (5) Output unified diagnostic features The final output is the fused multimodal features:

[0150] in, As a feature of fusion, This is the final layer of feature representation after hierarchical self-attention fusion. It contains comprehensive information from four modalities, which can be directly used as input for diagnostic decisions, and is applied to the prediction and analysis of papillary thyroid carcinoma.

[0151] S5: Input the fused features into the diagnostic network for prediction to obtain the disease diagnosis result.

[0152] In this embodiment, after completing the multimodal feature fusion, the fused features are used to predict the diagnosis of papillary thyroid carcinoma, and interpretability information of the model output is provided to assist doctors in clinical decision-making.

[0153] Fusion features Input diagnostic network:

[0154] in, This is the predicted probability value for papillary thyroid carcinoma. This refers to a fully connected neural network or other classification model.

[0155] Based on predicted probability Set threshold :

[0156] in, For the final diagnosis, It is positive. It was negative.

[0157] To enhance clinical trust, interpretability analysis is provided for the model's prediction results: Feature importance analysis: Calculating the contribution of each modality and region in the fused features:

[0158] in, For the first The magnitude of the influence of each modality on the diagnostic results.

[0159] Example 2 This embodiment discloses a disease diagnosis system based on multimodal spatial-frequency domain adaptive fusion, including: The standardization module is configured to acquire multimodal data and perform standardization processing on each data to obtain uniformly standardized multimodal data. The feature alignment module is configured to: encode the unified and standardized multimodal data separately to obtain the initial feature representations of the multimodal data; perform projection and gating alignment and cross-modal interactive attention alignment on the initial feature representations of each modality to obtain the enhanced representations of each modality; and then fuse the enhanced representations of each modality into a shared feature representation. The feature enhancement module is configured to: extract deep features from the enhanced representation of each modality to obtain deep features of each modality, and then obtain multi-domain enhanced features of each modality after adaptive multi-domain feature enhancement processing; The feature fusion module is configured to: semantically align the multi-domain enhanced features of each modality and then fuse them through a hierarchical attention mechanism to obtain fused features; The identification and diagnosis module is configured to input fused features into the diagnosis network for prediction to obtain disease diagnosis results.

[0160] Example 3 The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method of Embodiment 1.

[0161] Example 4 The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method of Embodiment 1.

[0162] The steps and methods involved in the apparatuses of Embodiments 3 and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0163] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0164] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0165] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A disease diagnosis method based on multimodal spatial-frequency domain adaptive fusion, characterized in that, include: Acquire multimodal data and perform standardization processing on each data point to obtain unified and standardized multimodal data; The standardized multimodal data are encoded separately to obtain the initial feature representation of the multimodality; After projecting and gating the initial feature representations of each modality and performing cross-modal interactive attention alignment, the enhanced representations of each modality are obtained, and then the enhanced representations of each modality are fused into a shared feature representation. Deep feature extraction is performed on the enhanced representation of each modality to obtain the deep features of each modality. After adaptive multi-domain feature enhancement processing, the multi-domain enhanced features of each modality are obtained. After semantic alignment of the multi-domain enhancement features of each modality, they are fused through a hierarchical attention mechanism to obtain fused features; The fused features are input into the diagnostic network for prediction, resulting in a disease diagnosis.

2. The disease diagnosis method based on multimodal spatial-frequency domain adaptive fusion as described in claim 1, characterized in that, The multimodal data includes single-cell transcriptome data, hyperspectral images, text data, and video data.

3. The disease diagnosis method based on multimodal spatial-frequency domain adaptive fusion as described in claim 2, characterized in that, The specific steps to obtain unified and standardized multimodal data are as follows: A cell adjacency graph was constructed based on single-cell transcriptome data. After clustering the gene sets, several gene blocks were obtained. Each gene block was smoothed and normalized on the cell adjacency graph. The smoothed gene blocks were corrected based on the local bias correction factor to obtain the standardized expression of each gene block. After merging, a single-cell expression matrix was obtained. After the hyperspectral image is divided into spatial blocks, multiple spatial blocks are obtained. The channels within each spatial block are normalized based on the local statistics and block offset of each spatial block. The normalized spatial blocks are corrected based on the local statistics and spectral features of the reconstruction residual to obtain the corrected spatial blocks. After merging, a normalized hyperspectral cube is obtained. Entities are identified from text data and their corresponding concepts are obtained. A medical semantic graph is constructed and then input into a graph neural network for encoding to obtain a structured text representation. Frame-level feature extraction is performed on video data to obtain frame feature vectors. Candidate keyframes are selected from the frame feature vectors based on the comprehensive significance score. Redundancy is removed from the corrected candidate keyframes to obtain the final keyframes. The final keyframes are then encoded into frame-level feature sequences.

4. The disease diagnosis method based on multimodal spatial-frequency domain adaptive fusion as described in claim 1, characterized in that, After projecting and gating the initial feature representations of each modality, and then performing cross-modal interactive attention alignment, the enhanced representations of each modality are obtained as follows: The initial feature representations of each modality are mapped to a unified dimension to obtain the feature representations of each modality; the feature representations of each modality are weighted based on the modality gating vector to obtain the weighted feature representations. Cross-modal attention is computed on the weighted feature representation to obtain the intermodal interaction representation; based on the intermodal interaction representation and the weighted feature representation, the enhanced representation of each modality is obtained.

5. The disease diagnosis method based on multimodal spatial-frequency domain adaptive fusion as described in claim 1, characterized in that, Using composite loss: in, For compound loss, , , , For hyperparameters, For modal reconstruction loss, For cross-modal contrast loss, To align the regularization loss, For parameter regularization terms.

6. The disease diagnosis method based on multimodal spatial-frequency domain adaptive fusion as described in claim 1, characterized in that, After adaptive multi-domain feature enhancement processing, the multi-domain enhanced features of each modality are obtained as follows: define a set of domain features and set domain attention weights; perform domain enhancement on the deep features of each modality based on the domain attention weights to obtain the multi-domain enhanced features of each modality.

7. The disease diagnosis method based on multimodal spatial-frequency domain adaptive fusion as described in claim 1, characterized in that, The specific fusion features obtained are as follows: The multi-domain enhancement features of each modality are projected onto a unified embedding space to obtain a unified embedding vector for each modality. The unified embedding vectors of each modality are merged to obtain the unified embedding feature; Multi-layer self-attention operation is performed on the unified embedding features to obtain fused features.

8. A disease diagnosis system based on multimodal spatial-frequency domain adaptive fusion, characterized in that, include: The standardization module is configured to acquire multimodal data and perform standardization processing on each data to obtain uniformly standardized multimodal data. The feature alignment module is configured to encode the standardized multimodal data separately to obtain the initial feature representation of the multimodality. After projecting and gating the initial feature representations of each modality and performing cross-modal interactive attention alignment, the enhanced representations of each modality are obtained, and then the enhanced representations of each modality are fused into a shared feature representation. The feature enhancement module is configured to: extract deep features from the enhanced representation of each modality to obtain deep features of each modality, and then obtain multi-domain enhanced features of each modality after adaptive multi-domain feature enhancement processing; The feature fusion module is configured to: semantically align the multi-domain enhanced features of each modality and then fuse them through a hierarchical attention mechanism to obtain fused features; The identification and diagnosis module is configured to input fused features into the diagnosis network for prediction to obtain disease diagnosis results.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the disease diagnosis method based on multimodal spatial frequency domain adaptive fusion as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the disease diagnosis method based on multimodal spatial frequency domain adaptive fusion as described in any one of claims 1-7.

Images