Auxiliary diagnosis method based on label-aware disentangled multi-modal graph learning

Abstract

This invention relates to the field of data processing technology, specifically providing an auxiliary diagnostic method based on label-aware de-entanglement multimodal graph learning. The method includes: acquiring a multimodal medical dataset containing multi-image modal features, non-image data, and disease labels; de-entanglement processing the image data to separate cross-modal shared features and modality-specific features, fusing them into image fusion features; semantically aligning these features with the non-image data and using disease labels for comparative learning to generate discriminative initial joint features; subsequently using these features as initial features for graph nodes to construct a graph neural network that dynamically calculates adjacency relationships based on node features; and finally, using graph convolution for feature propagation and representation optimization to achieve high-precision classification and prediction of brain diseases. This invention solves the problems of excessive graph structure-irrelevant noise, multimodal information aliasing, and cross-modal semantic misalignment in existing methods, improving the accuracy and robustness of brain disease diagnosis.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to an auxiliary diagnostic method based on label-aware unentangled multimodal graph learning. Background Technology

[0002] Accurate diagnosis and early prediction of brain diseases such as Alzheimer's disease and autism spectrum disorder are crucial for clinical intervention and disease management. With the development of neuroimaging technologies, multimodal data (such as structural magnetic resonance imaging (sMRI), functional magnetic resonance imaging (fMRI), and positron emission tomography (PET)) have become key sources for discovering disease biomarkers. However, single-modal information has limitations and cannot fully reflect the complex pathological mechanisms of brain diseases. Therefore, effectively integrating multimodal information to improve diagnostic accuracy has become a key challenge in current research.

[0003] Graph neural networks, with their powerful relational modeling capabilities, have been introduced into brain disease prediction tasks. By constructing subjects as graph nodes and utilizing population relationships for feature propagation, they have improved model performance to some extent. However, existing methods have significant shortcomings: First, graph structure construction is often based on non-disease-related features such as age and location, or simple feature similarity, failing to explicitly incorporate disease label information. This results in a large number of spurious connections unrelated to the disease in the graph, interfering with model learning. Second, multimodal feature fusion often employs simple concatenation or weighted summation, failing to effectively decouple cross-modal shared information from modality-specific information, leading to information aliasing and feature redundancy, weakening the model's discriminative and generalization abilities. Furthermore, existing methods typically employ static or weakly adaptive graph structures, making it difficult to characterize the dynamic relationships in disease progression, and multi-stage separation optimization easily leads to error accumulation.

[0004] Therefore, there is an urgent need for a new technical solution that can achieve multimodal feature deentanglement, label perception map construction, and end-to-end dynamic optimization to overcome the above-mentioned defects and promote the development of intelligent assisted diagnosis of brain diseases towards a more accurate, robust, and clinically practical direction. Summary of the Invention

[0005] To address the aforementioned shortcomings of existing technologies, this invention provides an auxiliary diagnostic method based on label-aware unentangled multimodal graph learning to solve the aforementioned technical problems.

[0006] In a first aspect, the present invention provides an auxiliary diagnostic method based on label-aware unentangled multimodal graph learning, comprising:

[0007] Obtain a multimodal medical dataset, which includes feature data of multiple image modalities, non-image data, and corresponding disease category labels; The feature data of the multiple image modalities are de-entangled to obtain the image fusion features of each subject. The image fusion features include cross-modal shared features and modality-specific features. The image fusion features are semantically aligned with the non-image data, and the disease category labels are used for comparative learning to generate discriminative initial joint features. Using the initial joint features as the initial features of the graph nodes, a graph neural network is constructed, wherein the adjacency relationship of the graph is dynamically calculated based on the features of the nodes. The graph neural network is used to perform feature propagation and node representation update to obtain optimized node representations. Based on the optimized node representations, disease classification prediction is performed and the prediction results are output.

[0008] In one optional implementation, a multimodal medical dataset is acquired, comprising: the multimodal medical dataset being a dataset constructed for the diagnosis of brain diseases, wherein the imaging modalities include at least structural magnetic resonance imaging and functional magnetic resonance imaging, and the disease category labels include at least one of Alzheimer's disease, autism spectrum disorder, schizophrenia, mild cognitive impairment, and vascular dementia.

[0009] In an optional implementation, the feature data of the multiple image modalities are de-entangled to obtain the image fusion features of each subject, including: The features of each image modality are mapped to a unified latent space to obtain the projection features of each modality; Based on the cross-modal attention mechanism, the projected features of each modality are weighted and fused to generate cross-modal shared features; Modality-specific features are extracted from the projection features using modality-independent encoders for each modality. The cross-modal shared features are combined with the modality-specific features to form the image fusion features for each subject.

[0010] In an optional implementation, the image fusion features are semantically aligned with the non-image data, and contrastive learning is performed using the disease category labels to generate discriminative initial joint features, including: The non-image data is encoded to obtain a non-image feature representation; By employing a cross-modal attention mechanism, the image fusion features are semantically aligned with the non-image feature representations to generate an initial joint feature representation; Using the disease category labels corresponding to known subjects in the multimodal medical dataset as supervision signals, a contrastive learning loss function is employed to narrow the distance between samples with the same disease category label and widen the distance between samples with different disease category labels in the feature space formed by the joint feature representation, thereby optimizing the discriminativeness of the joint feature representation.

[0011] In an optional implementation, using the disease category labels corresponding to known subjects in the multimodal medical dataset as the supervision signal, a contrastive learning loss function is employed to shorten the distance between samples with the same disease category label and widen the distance between samples with different disease category labels in the feature space constituted by the joint feature representation, including: Calculate the similarity between any two sample features in the initial joint feature representation; For each sample, in the feature space, other samples with the same disease category label as the sample are considered as positive sample pairs, and other samples with different disease category labels as the sample are considered as negative sample pairs; By minimizing the contrastive learning loss function, the feature space is optimized, thereby increasing the feature similarity between positive sample pairs and decreasing the feature similarity between negative sample pairs.

[0012] In an optional implementation, the generation of the disease category label includes at least one of the following steps: Subjects are classified into diseases using clinical diagnostic criteria to obtain disease category labels; The subjects were staged based on neuropsychological scale scores to obtain disease category labels; The subjects were classified into disease subtypes using neuroimaging biomarkers to obtain disease category labels.

[0013] In an optional implementation, a graph neural network is constructed using the initial joint features as the initial features of the graph nodes, including: The initial joint features are used as the initial feature matrix of the graph nodes; In each layer of the graph neural network, the similarity between nodes is dynamically calculated based on the feature representations of all nodes in the current layer, and the adjacency matrix of the current layer is constructed based on the similarity.

[0014] In an optional implementation, feature propagation and node representation updating are performed through the graph neural network to obtain optimized node representations. Disease classification prediction is then performed based on these optimized node representations, and the prediction results are output, including: Based on the adjacency matrix of the current layer, the feature information of the neighboring nodes is aggregated through graph convolution operations, and the node feature matrix of the next layer is updated to obtain the feature matrix of the next layer. After multiple iterations, the node feature matrix of the last layer is output as the optimized node representation. The optimized node representation is input into the classifier, which then predicts the disease category of each subject corresponding to each node and outputs the probability distribution or category label of each subject belonging to different disease categories.

[0015] In an optional implementation, it further includes: An end-to-end optimization is performed on the unentanglement module, the contrastive learning module, and the graph neural network using a joint loss function that includes classification loss, contrastive learning loss, and orthogonal clustering constraint loss.

[0016] Secondly, a device is provided, comprising: The memory is used to store auxiliary diagnostic programs based on label-aware unentangled multimodal graph learning; The processor is configured to implement the steps of the label-aware, unentangled multimodal graph learning-based auxiliary diagnostic method as provided in the first aspect when executing the auxiliary diagnostic program based on label-aware, unentangled multimodal graph learning.

[0017] Thirdly, a computer-readable storage medium is provided, on which an auxiliary diagnostic program based on label-aware unentangled multimodal graph learning is stored. When the program is executed by a processor, it implements the steps of the auxiliary diagnostic method based on label-aware unentangled multimodal graph learning provided in the first aspect.

[0018] The beneficial effects of this invention are as follows: the label-aware unentangled multimodal graph learning-based auxiliary diagnostic method provided by this invention has the following significant advantages compared with the prior art: First, by introducing adversarial learning and orthogonal constraints, deep deentanglement of multimodal features is achieved, effectively separating the common disease features shared across modalities from the specific features of each modality, avoiding information overlap and redundancy, and significantly improving the discriminative power of features and the generalization ability of the model.

[0019] Second, a dynamic graph construction mechanism based on label-aware contrastive learning is proposed. During the training phase, known disease labels are used to optimize the feature space, so that similar samples naturally cluster in the feature space. Then, adjacency relationships are constructed in the graph neural network based on dynamic feature similarity. This not only realizes the implicit perception of disease relevance by the graph structure, but also ensures the applicability of the inference phase, fundamentally eliminating the information interference caused by false connections.

[0020] Third, an end-to-end joint optimization framework is adopted, which integrates feature deentanglement, semantic alignment, dynamic graph learning and disease classification prediction. This enables collaborative updating of parameters across multiple modules, avoids error accumulation in staged optimization, enhances the overall stability and repeatability of the model, and better meets the actual needs of clinical diagnosis for algorithm robustness. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0023] Figure 2 This is another illustrative flowchart of a method according to an embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0027] The method provided in this embodiment of the invention is executed by a computer device, and correspondingly, the auxiliary diagnostic system based on label-aware unentangled multimodal graph learning runs in the computer device.

[0028] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The implementing entity can be an auxiliary diagnostic system based on label-aware unentangled multimodal graph learning. Depending on different requirements, the order of steps in this flowchart can be changed, and some can be omitted.

[0029] like Figure 1 As shown, the method includes: S1. Obtain a multimodal medical dataset, which includes feature data of multiple image modalities, non-image data, and corresponding disease category labels; S2. The feature data of the multiple image modalities are de-entangled to obtain the image fusion features of each subject. The image fusion features include cross-modal shared features and modality-specific features. S3. Semantically align the image fusion features with the non-image data, and use the disease category labels for comparative learning to generate discriminative initial joint features; S4. Using the initial joint features as the initial features of the graph nodes, construct a graph neural network, wherein the adjacency relationship of the graph is dynamically calculated based on the features of the nodes; S5. Perform feature propagation and node representation update through the graph neural network to obtain optimized node representations, and perform disease classification prediction based on the optimized node representations to output the prediction results.

[0030] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0031] A multimodal medical dataset suitable for the diagnosis of brain diseases needs to be acquired and constructed. Specifically, the dataset should contain multidimensional information from multiple subjects, mainly covering the following three types of data: S101. Imaging Modal Data: This includes at least two core imaging modalities: structural magnetic resonance imaging (sMRI) and functional magnetic resonance imaging (fMRI). sMRI data is used to extract structural features such as brain region volume and cortical thickness; fMRI data is used to calculate the brain functional connectivity matrix, reflecting the coordinated activity patterns between different brain regions. Furthermore, depending on available clinical resources, positron emission tomography (PET) data can be further incorporated to obtain information on brain metabolism or amyloid deposition.

[0032] S102. Non-imaging data: This includes demographic and clinical information for each subject, such as age, sex, years of education, data collection site code, and Mini-Mental State Examination (MMSE) score.

[0033] S103. Disease Category Labels: Labels are derived from professional clinical diagnosis or standardized assessment procedures and are used to indicate the disease status of each subject. The labels must cover at least one or more of Alzheimer's disease (AD), autism spectrum disorder (ASD), schizophrenia (SZ), mild cognitive impairment (MCI), and vascular dementia (VaD) to achieve binary or multi-class prediction tasks. For example, in a typical Alzheimer's disease research dataset, labels may include three categories: "healthy control (CN)," "mild cognitive impairment (MCI)," and "Alzheimer's disease (AD)."

[0034] After data acquisition, standardized preprocessing is required: sMRI images are registered and segmented to extract brain region features; fMRI data undergoes head motion correction and frequency domain filtering before functional connectivity calculation; non-image data is normalized or one-hot encoded. Ultimately, each subject is represented as a set of multimodal feature vectors and their corresponding disease labels.

[0035] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0036] (1) For the feature matrix of each mode (N is the number of subjects, A linear projection is performed on the feature dimension of the m-th modality, mapping it to a d-dimensional unified latent space:

[0037] in Let be the projected feature of the m-th mode.

[0038] A scaled dot product attention mechanism is used to model cross-modal dependencies and calculate inter-modal similarity:

[0039] All modal features are fused through attention-weighted fusion, followed by a linear layer. Generate shared representation

[0040] (2) Modality-specific representation extraction To preserve the unique information of each modality, a modality-specific representation is extracted using a dedicated multilayer perceptron (MLP):

[0041] in It is a dedicated encoder for the m-th modality, with independent parameters, ensuring the capture of modality-dependent characteristics.

[0042] Orthogonal Clustering Constraints (OCM): To achieve orthogonal separation between shared and modality-specific subspaces, an orthogonal clustering module is introduced. This module reduces feature redundancy through cluster distribution alignment and orthogonal constraints. Trainable cluster centers are defined. (K is the number of clusters), calculate the soft assignment probability of features and cluster centers based on the t-distribution:

[0043] in For degrees of freedom parameters, Let be the fusion feature of the i-th subject.

[0044] Construct the target distribution sharpening assignment result:

[0045] The orthogonal clustering loss is defined as:

[0046] By minimizing this loss, the orthogonality between the shared representation and the modality-specific representation is ensured, thereby enhancing the feature discriminative power.

[0047] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0048] Step S301: Non-image feature encoding Non-image data (such as age, gender, years of education, and data collection site) is first mapped to the same latent dimension as image features through a linear projection layer, resulting in a preliminary non-image feature matrix. To further capture the dependencies between features, a self-attention mechanism can be introduced for enhanced encoding, yielding the final non-image feature representation Hnon.

[0049] Non-image data (such as age, gender, and data collection site) Linear projection mapping to d-dimensional latent space:

[0050] Capturing internal dependencies of non-image features through a self-attention mechanism: .

[0051] Step S302: Cross-modal semantic alignment Employing a cross-attention mechanism to achieve image fusion of Himg features and non-image features Semantic alignment. Specifically, with As a query, As keys and values, attention weights are calculated and image features are weighted and fused to generate an initial joint feature representation:

[0052] This process aligns information from different modalities in a unified semantic space, providing a foundation for subsequent comparative learning.

[0053] Step S303: Comparative Learning Optimization Using the disease category labels of known subjects in the dataset as supervision signals, a contrastive learning method is employed to optimize the joint feature space.

[0054] The specific implementation is as follows: Similarity calculation: For the joint feature representation Z (0) Any two sample features and Calculate the cosine similarity: Positive and negative sample construction: For each sample i, form a positive sample pair set P(i) with all samples with the same disease label; form a negative sample pair set N(i) with all samples with different disease labels.

[0055] Loss function optimization: Normalized temperature-scaled cross-entropy loss (NT-Xent) is used as the contrastive learning loss function.

[0056] in For temperature hyperparameters, For feature similarity, The indicator function is used. The total module loss is the mean of the contrastive losses for all similar sample pairs:

[0057] in In order to be with the first A set of samples with the same sample label.

[0058] By minimizing this loss, the model brings positive sample pairs closer together and pulls negative sample pairs further apart in the feature space, thereby enhancing the disease discriminative power of the features.

[0059] In real-world clinical datasets, disease category labels can be generated using one or more authoritative methods, such as: Clinical diagnostic criteria: Subjects are classified into diseases according to internationally recognized diagnostic guidelines (such as DSM-5, NINCDS-ADRDA, etc.) and labeled with categories such as "AD", "CN", and "ASD".

[0060] Neuropsychological scale scoring: Based on the scoring results of cognitive assessment scales such as MMSE, MoCA, and ADAS-Cog, the subjects are classified into disease stages (such as "mild cognitive impairment" or "moderate dementia").

[0061] Neuroimaging biomarkers: Using imaging biomarkers (such as Aβ-PET positivity, tau-PET positivity, degree of hippocampal atrophy, etc.) to classify subjects into disease subtypes (such as "AD tau positive type", "vascular dementia type", etc.).

[0062] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0063] Graph neural networks employ a dynamic graph construction mechanism, where adjacency relationships are no longer fixed but are dynamically generated based on the real-time similarity of node features.

[0064] S401. Node Feature Initialization The initial joint feature representation obtained after semantic alignment and contrastive learning optimization The initial feature matrix serves as the graph node matrix, where N is the number of subjects and d is the feature dimension. Each subject corresponds to a node in the graph.

[0065] S402. Calculation of Dynamic Adjacency Matrix at Each Level In each layer l (l=0,1,…,L) of the graph neural network In step 1), based on the node feature matrix H of the current layer... (l) Dynamically calculate the similarity between nodes and construct the adjacency matrix A of the current layer. (l) The specific steps are as follows: Similarity calculation: For any two nodes i and j, calculate the cosine similarity between their feature vectors hi(l) and hj(l): Adjacency matrix construction: To enhance the sparsity of the graph and reduce noisy connections, an adjustable similarity threshold is set. (Typically, the value is between 0.5 and 0.8), and only connections with a similarity higher than the threshold are retained: This construction method does not rely on node labels, but is based solely on feature similarity, making it perfectly suitable for both the training and inference phases.

[0066] S403. Normalization Process To maintain numerical stability, the adjacency matrix is... After adding a self-loop, perform symmetric normalization: in For degree matrix, .

[0067] Graph convolutional feature propagation, based on dynamically constructed normalized adjacency matrix. Feature propagation and updating are performed using graph convolution operations: in Let σ be the learnable weight matrix of the ll-th layer, and σ be the activation function (such as ReLU).

[0068] in Embed for the initial node. Let be the weight matrix of the l-th layer.

[0069] In one embodiment of the present invention, based on step S5, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.

[0070] By utilizing the defined graph convolutional layers, the final node representation is obtained after multiple graph convolutions. .

[0071] The predicted probability is output through the MLP classifier:

[0072] in , Here, C represents the classifier parameters, and C is the number of disease categories (e.g., C=2 for binary classification of healthy / patient, C=3 for triadic classification of CN / MCI / AD). The Softmax function converts the scores into a probability distribution, representing the posterior probability that each subject belongs to a different category.

[0073] The final output of the model can be provided in two forms depending on the actual application requirements: Probability distribution output: Direct output Each row represents the probability of a subject belonging to a category, which is suitable for scenarios that require risk assessment or uncertainty analysis.

[0074] Category label output: Take the category with the highest probability in each row as the predicted label, that is:

[0075] Suitable for directly assisting clinical classification decisions.

[0076] Based on the above embodiments, in one embodiment, end-to-end optimization is performed on the deentanglement module processing module, the contrastive learning module, and the graph neural network through a joint loss function that includes classification loss, contrastive learning loss, and orthogonal clustering constraint loss.

[0077] The total loss of the model is a weighted sum of the classification loss, label-aware contrastive loss, and orthogonal clustering loss:

[0078] in For cross-entropy loss, , To balance the hyperparameters of each loss term and achieve synergistic optimization of feature learning, graph construction, and prediction performance.

[0079] Please refer to Figure 2 The workflow of the label-aware unentangled multimodal graph learning-based auxiliary diagnostic method includes: 1. Data Input and Preprocessing: Input a multimodal medical dataset, covering M image modal features. (e.g., MRI, fMRI, PET, etc., each modality feature matrix) N is the number of subjects. For the corresponding modal feature dimension), non-image data D (including demographic information such as age, gender, and collection site), , The input data includes non-image feature dimensions and corresponding disease labels Y (such as AD / CN / sMCI classification labels for Alzheimer's disease, and ASD / NC binary classification labels for autism). The input data undergoes standardized preprocessing to eliminate dimensional differences between different modal features, while samples with missing values ​​exceeding a threshold are removed to ensure data quality.

[0080] 2. Multimodal Feature Deentanglement Processing: The Image Modal Representation Deentanglement Module (IMRD) initiates the processing flow, first mapping each modal feature to a d-dimensional unified latent space through a linear projection layer to obtain the projected features. Then, a scaled dot product attention mechanism is used to model cross-modal dependencies and fuse them to generate a shared representation that captures the commonalities of diseases. Simultaneously, a dedicated MLP encoder is used to extract the unique specific representations of each modality. Finally, the soft assignment probability is calculated using orthogonal clustering constraints (OCM). With target distribution By minimizing the orthogonal clustering loss This achieves orthogonal separation between shared subspaces and modality-specific subspaces, avoiding information aliasing.

[0081] 3. Label-Aware Graph Structure Construction: The Label-Aware Contrast Module (CLAM) receives the feature representation output by IMRD and preprocessed non-image data. It first performs linear projection and self-attention encoding on the non-image data to obtain the non-image feature embedding. Then, a cross-attention mechanism is used to achieve non-image embedding and image feature integration. Semantic alignment is used to generate a joint feature representation Z; subsequently, disease labels are used as supervision signals, and label-perceptual contrastive loss is applied. Optimize the feature distance between similar and dissimilar samples, and construct a label-aware adjacency matrix based on the learned feature similarity. Establish valid connections only among subjects of the same type, and eliminate irrelevant spurious connections.

[0082] 4. Graph Convolution Feature Propagation and Representation Optimization: The prediction module based on GNN calls the constructed adjacency matrix. With the joint feature representation Z, first construct the adjacency matrix with self-loops. Then, symmetric normalization is performed to obtain the normalized adjacency matrix. Use joint feature representations as initial embeddings for graph nodes. Input a multi-layer graph convolutional network, and then... The inter-layer propagation mechanism aggregates neighboring node information and dynamically updates node representations; after L layers of graph convolution operations, the optimized final node representation is output. .

[0083] 5. Disease Prediction and Model Optimization: Characterizing the Final Node Input an MLP classifier and output the disease prediction probability for each subject through the Softmax function. ;Calculate the cross-entropy loss between the predicted result and the true label Y. Combined with label perception contrast loss With orthogonal clustering loss , constituting the total loss (enter, (For loss balancing hyperparameters); use the Adam optimizer to backpropagate and update all trainable parameters in the framework (including projection layer weights, attention mechanism parameters, MLP encoder parameters, cluster centers, graph convolutional layer weights, etc.), iterate training until the total loss converges, and output the final disease prediction result and the trained model.

[0084] Figure 3The label-aware unentangled multimodal graph learning-based auxiliary diagnostic method provided in this application embodiment can be applied to devices. Those skilled in the art will understand that the device structures involved in the embodiments of this invention do not constitute a limitation on the device. A device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the device includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0085] The device 300 may include a processor 310, a memory 320, and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0086] The memory 320 can be used to store execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 is able to perform some or all of the steps in the above method embodiments.

[0087] The processor 310 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0088] The communication unit 330 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.

[0089] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0090] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0091] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0092] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0093] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0095] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. An auxiliary diagnosis method based on label-aware disentangled multi-modal graph learning, characterized in that, include: Obtain a multimodal medical dataset, which includes feature data of multiple image modalities, non-image data, and corresponding disease category labels; The feature data of the multiple image modalities are de-entangled to obtain the image fusion features of each subject. The image fusion features include cross-modal shared features and modality-specific features. The de-entanglement module, contrastive learning module, and graph neural network are optimized end-to-end using a joint loss function that includes classification loss, contrastive learning loss, and orthogonal clustering constraint loss. The soft assignment probability and target distribution are calculated through orthogonal clustering constraints, and the orthogonal separation of cross-modal shared features and modality-specific features is achieved through orthogonal clustering constraint loss. The image fusion features are semantically aligned with the non-image data, and comparative learning is performed using the disease category labels to generate discriminative initial joint features, including: The non-image data is encoded to obtain a non-image feature representation; By employing a cross-modal attention mechanism, the image fusion features are semantically aligned with the non-image feature representations to generate an initial joint feature representation; Using the disease category labels of known subjects in the multimodal medical dataset as supervision signals, a contrastive learning loss function is adopted to narrow the distance between samples with the same disease category label and widen the distance between samples with different disease category labels in the feature space formed by the joint feature representation, so as to optimize the discriminativeness of the joint feature representation. Using the initial joint features as the initial features of the graph nodes, a graph neural network is constructed, wherein the adjacency relationship of the graph is dynamically calculated based on the features of the nodes. The graph neural network is used to perform feature propagation and node representation update to obtain optimized node representations. Based on the optimized node representations, disease classification prediction is performed and the prediction results are output.

2. The method of claim 1, wherein, The multimodal medical dataset is a dataset constructed for the diagnosis of brain diseases. Its imaging modalities include at least structural magnetic resonance imaging and functional magnetic resonance imaging. The disease category labels include at least one of Alzheimer's disease, autism spectrum disorder, schizophrenia, mild cognitive impairment, and vascular dementia.

3. The method of claim 1, wherein, The feature data of the multiple image modalities are de-entangled to obtain the image fusion features of each subject, including: The features of each image modality are mapped to a unified latent space to obtain the projection features of each modality; Based on the cross-modal attention mechanism, the projected features of each modality are weighted and fused to generate cross-modal shared features; Modality-specific features are extracted from the projection features using modality-independent encoders for each modality. The cross-modal shared features are combined with the modality-specific features to form the image fusion features for each subject.

4. The method of claim 1, wherein, Using the disease category labels corresponding to known subjects in the multimodal medical dataset as supervision signals, and employing a contrastive learning loss function, in the feature space constituted by the joint feature representation, the distance between samples with the same disease category label is reduced, while the distance between samples with different disease category labels is increased, including: Calculate the similarity between any two sample features in the initial joint feature representation; For each sample, in the feature space, other samples with the same disease category label as the sample are considered as positive sample pairs, and other samples with different disease category labels as the sample are considered as negative sample pairs; By minimizing the contrastive learning loss function, the feature space is optimized, thereby increasing the feature similarity between positive sample pairs and decreasing the feature similarity between negative sample pairs.

5. The method of claim 4, wherein, The generation of the disease category label includes at least one of the following steps: Subjects are classified into diseases using clinical diagnostic criteria to obtain disease category labels; The subjects were staged based on neuropsychological scale scores to obtain disease category labels; The subjects were classified into disease subtypes using neuroimaging biomarkers to obtain disease category labels.

6. The method of claim 1, wherein, Using the initial joint features as the initial features of the graph nodes, a graph neural network is constructed, including: The initial joint features are used as the initial feature matrix of the graph nodes; In each layer of the graph neural network, the similarity between nodes is dynamically calculated based on the feature representations of all nodes in the current layer, and the adjacency matrix of the current layer is constructed based on the similarity.

7. The method of claim 6, wherein, The graph neural network performs feature propagation and node representation updates to obtain optimized node representations. Based on these optimized node representations, disease classification prediction is performed, and the prediction results are output, including: Based on the adjacency matrix of the current layer, the feature information of the neighboring nodes is aggregated through graph convolution operations, and the node feature matrix of the next layer is updated to obtain the feature matrix of the next layer. After multiple iterations, the node feature matrix of the last layer is output as the optimized node representation. The optimized node representation is input into the classifier, which then predicts the disease category of each subject corresponding to each node and outputs the probability distribution or category label of each subject belonging to different disease categories.

Images