Classification method and device based on graph neural network, computer device and medium

Abstract

The application discloses the above-mentioned classification method, device, equipment and storage medium based on the graph neural network, the method implementation, including: obtaining the MRI image to be detected and the text information to be detected corresponding to the MRI image to be detected, and generating the PET image to be detected based on the MRI image to be detected; the MRI image to be detected and the PET image to be detected are converted into MRI graph structure and PET graph structure respectively, wherein the MRI graph structure and the PET graph structure comprise node features and edge features respectively; based on the node features and the edge features, the image features are obtained; based on the text information to be detected, the text features are obtained; the image features and the text features are fused, and based on the fusion features, the classification results are obtained. Through the cross-modal fusion of the text mode and the image mode, the influence of the missing mode information can be made up, the potential pathological features can be mined, the multi-source complementary information can be integrated, the model robustness can be enhanced, and strong support is provided for early diagnosis of Alzheimer's disease and other neurodegenerative diseases.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to a classification method, apparatus, computer device and medium based on graph neural networks. Background Technology

[0002] Alzheimer's disease (AD) is a progressive neurodegenerative disease, usually beginning with mild cognitive impairment (MCI). Its irreversible pathological process and lack of effective treatments make early and accurate diagnosis crucial to slowing disease progression.

[0003] Currently, clinical diagnosis primarily relies on multimodal imaging data such as magnetic resonance imaging (MRI) and positron emission tomography (PET), combined with comprehensive assessment of patient clinical indicators (such as age and cognitive scores). However, this process requires manual prediction by neurosurgeons, consuming significant manpower and time, and the results are easily influenced by the doctor's experience and subjective factors. While machine learning technology has been initially applied to the early diagnosis of Alzheimer's disease (AD) with some success, it still has significant limitations. Existing methods generally lack deep fusion of cross-modal information when processing multimodal data, resulting in insufficient utilization of data complementarity. Furthermore, traditional feature extraction algorithms struggle to capture subtle pathological changes in the complex topology of the brain, leading to the loss of crucial diagnostic information. These technological bottlenecks directly restrict the accuracy and efficiency of early AD diagnosis, necessitating the exploration of more efficient, objective, and intelligent diagnostic solutions. Summary of the Invention

[0004] Therefore, it is necessary to provide a classification method, apparatus, computer device, and medium based on graph neural networks to address the above-mentioned technical problems and solve at least one of the problems existing in the prior art.

[0005] Firstly, a classification method based on graph neural networks is provided, including:

[0006] Acquire the MRI image to be detected and the corresponding text information to be detected, and generate the PET image to be detected based on the MRI image to be detected;

[0007] The MRI image and PET image to be detected are converted into MRI image structure and PET image structure, respectively, wherein the MRI image structure and PET image structure include node features and edge features, respectively.

[0008] Based on the node features and edge features, image features are obtained;

[0009] Based on the text information to be detected, text features are obtained;

[0010] The image features and text features are fused to obtain fused features, and a classification result is obtained based on the fused features.

[0011] In one possible implementation, obtaining image features based on the node features and edge features includes:

[0012] The node features are input into the graph neural network module for processing to obtain global features;

[0013] Based on the node features and edge features, the data are input into a message passing network for processing to obtain local features;

[0014] The image features are obtained based on the global and local features.

[0015] In one possible implementation, the step of inputting the node features into a graph neural network module for processing to obtain global features includes:

[0016] The node features are flattened to obtain the original node feature sequence;

[0017] According to the preset node priority strategy, the nodes in the original node feature sequence are reordered to obtain the target node feature sequence.

[0018] The global features are obtained based on the target feature node sequence.

[0019] In one possible implementation, reordering the nodes in the original node feature sequence according to a preset node priority strategy includes:

[0020] Calculate the node importance index for each node feature in the original node feature sequence;

[0021] The node features in the original node feature sequence are reordered according to the node importance index from low to high.

[0022] In one possible implementation, converting the MRI image and the PET image to be detected into MRI image structures and PET image structures, respectively, includes:

[0023] The MRI image and the PET image to be detected are segmented to obtain multiple image blocks, and the image blocks are used as nodes.

[0024] The edges between nodes are obtained based on spatial relationships;

[0025] The weight of each edge is determined based on the correlation between the endpoint nodes of each edge.

[0026] Based on the nodes, edges, and edge weights, the MRI image structure and PET image structure are obtained.

[0027] In one possible implementation, the feature fusion of the image features and text features to obtain fused features includes:

[0028] The image features are linearly transformed to obtain multiple calculation factors;

[0029] Attention calculations and linear transformations are performed on the text features and multiple computational factors to obtain the fused features.

[0030] In one possible implementation, the step of performing attention calculations and linear transformations on the text features and multiple computational factors to obtain the fused features includes:

[0031] The text features are subjected to layer normalization to obtain the layer-normalized text features.

[0032] An attention mechanism is used to calculate the text features after the layer normalization process and multiple calculation factors to obtain preliminary fused features.

[0033] The preliminary fused features are input into a multilayer perceptron for processing to obtain the final fused features.

[0034] Secondly, a classification device based on a graph neural network is provided, comprising:

[0035] The image acquisition unit is used to acquire the MRI image to be detected and the text information to be detected corresponding to the MRI image to be detected, and to generate the PET image to be detected based on the MRI image to be detected;

[0036] The graph construction unit is used to convert the MRI image to be detected and the PET image to be detected into MRI graph structure and PET graph structure, respectively, wherein the MRI graph structure and the PET graph structure include node features and edge features, respectively.

[0037] The image feature acquisition unit is used to obtain image features based on the node features and edge features;

[0038] The text feature acquisition unit is used to obtain text features based on the text information to be detected;

[0039] The feature fusion and classification unit is used to fuse the image features and text features to obtain fused features, and to obtain a classification result based on the fused features.

[0040] Thirdly, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer-readable instructions, implements the steps of the graph neural network-based classification method as described above.

[0041] Fourthly, a readable storage medium is provided, the readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the graph neural network-based classification method as described above.

[0042] The aforementioned classification method, apparatus, computer device, and medium based on graph neural networks include the following steps: acquiring a target MRI image and corresponding text information to be detected, and generating a target PET image based on the target MRI image; converting the target MRI image and the target PET image into MRI graph structures and PET graph structures, respectively, wherein the MRI graph structure and the PET graph structure include node features and edge features, respectively; obtaining image features based on the node features and edge features; obtaining text features based on the text information to be detected; fusing the image features and text features to obtain fused features, and obtaining a classification result based on the fused features. In the embodiments of this application, generating a target PET image based on the target MRI image can compensate for the impact of missing modality information. Converting the target MRI image and the target PET image into graph structures allows for a more intuitive presentation of the complex connections between brain regions. The transformed graph-structured data can be seamlessly integrated with graph neural networks, providing them with an input format that better aligns with the topological characteristics of brain networks. This helps the model learn brain network topology more efficiently, uncover potential pathological features, and provide strong support for the early diagnosis of neurodegenerative diseases such as Alzheimer's. Furthermore, in the modality fusion stage, deep fusion of image and text features is achieved by adjusting the attention mechanism. This integrates complementary information from multiple sources, enhances model robustness, and uncovers potential correlations between modalities, thereby improving classification accuracy and highly meeting the practical needs of comprehensive clinical diagnosis. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1This is a schematic diagram of the overall model architecture of a classification method based on graph neural networks in one embodiment of this application;

[0045] Figure 2 This is a flowchart illustrating a classification method based on a graph neural network in one embodiment of this application;

[0046] Figure 3 This is a schematic diagram of a module structure of a Mamba module in one embodiment of this application;

[0047] Figure 4 This is a schematic diagram of a module structure of the modal fusion module in one embodiment of this application;

[0048] Figure 5 This is a schematic diagram of a classification device based on a graph neural network in one embodiment of this application;

[0049] Figure 6 This is a schematic diagram of a computer device according to one embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] The graph neural network-based classification method provided in this embodiment can be applied to, for example, Figure 1The model architecture includes a generative model module, a graph Mmaba architecture, and an adaptive fusion module. First, the acquired MRI image to be detected is input into the generative model module. In this module, a 3D generative adversarial network (GAN) generates a PET image to be detected based on the MRI image, enhancing the details and biomarker information in the image modality and enriching the image details. Then, the MRI and PET images are input into the graph Mmaba architecture, which includes an Mmaba module and a message passing network. The MRI and PET images are constructed as graph structures, and embedding operations are performed on the points and edges of the graph structures to obtain point (node) feature embeddings and edge feature embeddings. The graph Mmaba module processes the point feature embeddings to obtain global features, and the message passing network processes these embeddings to obtain local features. Finally, the global and local features are fused using a multilayer perceptron (MLP) to obtain an image feature representation. Image representation and topological information can be learned from the image module. To better integrate image representations and text information, the text information can be organized into tabular information, and tabular feature representations can be extracted. Then, the image feature representations and the tabular feature representations extracted from the tabular data can be fused through an adaptive modality fusion module. Finally, the fused hybrid modality representation is input into the final classification head to obtain the classification result.

[0052] The tabular feature representation is obtained by organizing the text information corresponding to the MRI image to be detected into a table, and then extracting features using a feature extraction model. This could include a table of clinical information about the user being detected, such as gender, age, weight, symptoms, and disease stage.

[0053] In one embodiment, such as Figure 2 As shown, a classification method based on graph neural networks is provided, including the following steps:

[0054] In step S110, the MRI image to be detected and the text information to be detected corresponding to the MRI image to be detected are obtained, and a PET image to be detected is generated based on the MRI image to be detected;

[0055] The MRI image to be detected refers to a magnetic resonance imaging (MRI) image. This image utilizes the magnetic resonance phenomenon caused by hydrogen protons in the human body being excited by radio frequency pulses in a magnetic field. The resulting signal is then processed by a computer to reconstruct an image of the internal structure of the human body. It can clearly display structures such as gray matter, white matter, and ventricles of the brain, and is highly sensitive to observing changes in brain morphology and structure. It can help doctors detect brain abnormalities related to Alzheimer's disease, such as hippocampal atrophy and white matter lesions.

[0056] PET images, or Positron Emission Tomography (PET) images, are produced by injecting a radioactive tracer into the body. This tracer participates in metabolic processes, and the positrons produced during their decay annihilate electrons, generating gamma rays that are detected by a detector, thus generating an image reflecting metabolic activity. In Alzheimer's disease diagnosis, PET images often use tracers that specifically bind to certain lesions in the brain, such as β-amyloid plaques. This allows for direct visualization of metabolic changes in specific brain regions, helping to detect abnormal brain function in the early stages of the disease. Even before significant structural changes in the brain are observed, areas with reduced metabolism can be detected, providing crucial evidence for early diagnosis.

[0057] The text information to be tested refers to the clinical information of the user being tested, which may include text information such as gender, age, weight, symptoms, and disease stage. This information can be obtained through methods such as doctor diagnosis and patient data entry.

[0058] Specifically, the MRI image to be detected can be input into the generative model module, which contains a pre-built 3D generative adversarial network (GAN). This GAN consists of a generator and a discriminator. The generator structure adopts the traditional Unet structure, which includes two downsampling layers and two upsampling layers, with the middle layer being the bottleneck layer of the Vit model. The discriminator structure is the same as that of the PatchGAN model. The specific process of generating a PET image to be detected based on the MRI image is as follows: First, the input MRI sample image is preprocessed, including normalization and standardization. The corresponding PET sample image is also preprocessed in the same way. Then, based on the generator, the preprocessed MRI sample image is converted into a feature representation similar to a PET image. The discriminator is used to judge the difference between the generated similar PET image and the real PET image. Based on the output of the discriminator, the loss functions of the generator and the discriminator are calculated. For the generator, its loss function aims to make the generated image as close as possible to the real PET image to deceive the discriminator. The 3D generative adversarial network (GAN) was trained iteratively through multiple rounds. During training, the generator continuously improved the quality of the generated images, while the discriminator continuously improved its ability to judge the realism of the images. The two competed and cooperated with each other, making the generated PET images increasingly closer to real PET images. When a preset convergence condition was met, such as the number of iterations reaching a certain value, or the loss values ​​of the generator and discriminator decreasing to a specific range, the MRI image to be detected was input into the generator to generate the final PET image to be detected.

[0059] In step S120, the MRI image to be detected and the PET image to be detected are converted into MRI image structure and PET image structure, respectively, wherein the MRI image structure and the PET image structure include node features and edge features, respectively.

[0060] Optionally, the MRI image and PET image to be detected are respectively input into the Mmaba architecture for processing, such as... Figure 3 As shown, the graph neural network architecture (Graph Mmaba architecture) can include part a and part b. Part a is used to convert the MRI image and PET image to be detected into MRI graph structures and PET graph structures, respectively. Part b is used to obtain image features based on the MRI graph structure and PET graph structure. Since the original MRI image and PET image to be detected are large, a 3D CNN (convolutional) can be used to downsample the image. Then, the downsampled image is segmented into non-overlapping image blocks using a block-segmentation strategy. These image blocks are used as nodes, and a 26-neighborhood connection model is used to form edges between nodes based on spatial relationships. The edge weights are determined by calculating the Pearson correlation coefficient between nodes, thus obtaining graph structure data. The graph structure can effectively represent the topological information of the brain.

[0061] In step S130, image features are obtained based on the node features and edge features;

[0062] like Figure 1 As shown, embedding operations can be performed on node features and edge features separately to obtain node embedded features and edge embedded features, thereby converting the node features and edge features into a feature representation suitable for model processing. Then, the node embedded features can be input into the Mmaba module for processing to obtain global features. Simultaneously, the node embedded features and edge embedded features are input into a message passing network for processing to obtain local features. Finally, the global features output from the Mmaba module and the local features output from the message passing network are input into a multilayer perceptron (MLP) for fusion to obtain the image feature representation.

[0063] In step S140, text features are obtained based on the text information to be detected;

[0064] Optionally, clinical information of the users to be tested can be collected and organized into standardized tabular data based on clinical attributes, such as gender, age, weight, symptom type, and disease stage, with each row representing one piece of clinical information for the user. Then, the text categories can be converted into numerical representations. For example, gender can be encoded using binary variables, such as male as 1 and female as 0. For multi-category variables, one-hot encoding or word embedding can be used. For disease stages, such as Alzheimer's disease stages, they can be mapped to ordered numerical values ​​(mild → 1, moderate → 2, severe → 3). Continuous numerical values ​​(such as age, weight, and cognitive score) can be normalized or standardized. All processed fields are combined into a two-dimensional matrix, with each column corresponding to a feature, thus obtaining the text feature representation, i.e., the tabular feature representation.

[0065] In step S150, the image features and text features are fused to obtain fused features, and a classification result is obtained based on the fused features.

[0066] like Figure 4 As shown, the adaptive fusion module may include a traditional transformer layer and a multilayer perceptron (MLP). The traditional transformer layer may include an attention layer and a feedforward layer. First, the image feature representation F... I The data can be converted into three computational factors by a multilayer perceptron (MLP), namely: MLPs consist of multiple linear layers for feature transformation, mapping image features to a suitable space to prepare for subsequent cross-modal information injection. This provides the initial feature representation of the text. Layer normalization (LayerNorm) is performed to stabilize the numerical distribution and facilitate computation. Through... A linear transformation is performed on the input table features, and then the transformed table features are passed through an attention layer or a feedforward layer. It is used as a gating parameter in the calculation and is combined with the initial feature representation of the text. By adding image-related information into the text feature representation using an attention mechanism, an updated feature representation is obtained. Then, to Perform layer normalization again, and then... A linear transformation is performed, followed by a nonlinear transformation using a multilayer perceptron (MLP). As a gating parameter, image information is incorporated into the tabular features via an MLP (Multi-Level Processing) approach to obtain a further updated feature representation.

[0067] Finally, the hybrid representation learned after cross-modal information injection will be... The data is input into the classification head, which is a linear layer, to obtain the classification result. This result can include SMCI (Stable Mild Cognitive Impairment) and PMCI (Progressive Mild Cognitive Impairment). The classification process can be illustrated by the following formula:

[0068]

[0069] It should be noted that the above text features are... Figure 4 The table feature in the text, because the text is organized in a tabular format, therefore... Figure 4 The table is used to represent the data.

[0070] in, As a weighting factor, it is used to scale or weight text features so that image information can influence the transformation process of text features; This is a bias factor used to offset text features and adjust the reference point for the transformation. It is a gating factor used to control the intensity of information flow through multiplication operations. When it is close to 0, information transmission is suppressed, and when it is close to 1, information is allowed to pass through.

[0071] In the embodiments of this application, generating PET images based on MRI images to be detected can compensate for the impact of missing modality information. Converting the MRI and PET images to be detected into graph structures allows for a more intuitive presentation of the complex connections between brain regions. The converted graph structure data can be seamlessly integrated with graph neural networks, providing them with an input format more consistent with the topological characteristics of the brain network. This helps the model learn brain network topology information more efficiently, uncover potential pathological features, and provide strong support for the early diagnosis of neurodegenerative diseases such as Alzheimer's disease. Furthermore, in the modality fusion stage, deep fusion of image and text features is achieved by adjusting the attention mechanism. This integrates complementary information from multiple sources, enhances model robustness, and uncovers potential correlations between modalities, thereby improving classification accuracy and highly meeting the practical needs of comprehensive clinical diagnosis.

[0072] In one embodiment of this application, obtaining image features based on the node features and edge features includes:

[0073] The node features are input into the graph neural network module for processing to obtain global features;

[0074] Based on the node features and edge features, the data are input into a message passing network for processing to obtain local features;

[0075] The image features are obtained based on the global and local features.

[0076] Optionally, a graph Mamba architecture can be used for image feature generation, which may include a messaging network and Mamba modules. For example... Figure 1 As shown, embedding operations can be performed on node features to transform them into a feature representation suitable for model processing. This yields the node embedding features (…). Figure 1 (The point embedding features are shown). These point embedding features can then be input into the Mmaba module for processing. The Mmaba module embeds a Selective State-Space Model (SSM). First, the node embedding representations are input into a Convolutional Projection layer and a Gate Projection layer, respectively. The Convolutional Projection layer performs convolution operations and projection transformations, while the Gate Projection layer calculates a gate signal used to control the data flow. Then, the output of the Convolutional Projection is input into a Convolutional layer (Conv) for convolution operations. The Convolutional layer calculates by sliding the convolution kernel across the data, extracting local features. An activation function, such as ReLU, is used to perform a non-linear transformation on the output of the Convolutional layer. The SSM component then captures the dynamic changes and global dependencies of the feature data processed by the activation function over time or sequence. Finally, the features processed by the SSM component are cross-multiplied with the gate signal, and the output projection layer performs a projection transformation on the result of the cross-multiplication, mapping the features to the final output space to obtain the global features.

[0077] Simultaneously, edge features can be embedded to obtain edge embedding features. These node embedding features and edge embedding features are then input into a message passing network (MPNN), which focuses on capturing local neighborhood features. For example, for each node in the MPNN, a message can be computed using a learnable message function (usually a neural network) based on its own node embedding features, the edge embedding features of its connected edges, and the node embedding features of its neighbors. Each node can aggregate messages received from all its neighbors. Using the aggregated messages, the node's state is updated using an update function (such as a gated recurrent unit (GRU) or a long short-term memory (LSTM) network. Through multiple iterations, the range of local neighborhood feature capture can be continuously expanded, enabling nodes to effectively capture rich local neighborhood features.

[0078] The above process of extracting global and local features can be represented by the following formula:

[0079] X i+1i =MPNN(X i E i)+Mamba(X i ).

[0080] Among them, X i Represents node embedding, Mamba(X) i ) represents the global feature, E i Represents edge node embedding, MPNN(X) i E i ) represents a local feature.

[0081] Finally, the global and local features are input into a multilayer perceptron (MLP) for linear transformation to obtain the final image feature representation.

[0082] In one embodiment of this application, the step of inputting the node features into a graph neural network module for processing to obtain global features includes:

[0083] The node features are flattened to obtain the original node feature sequence;

[0084] According to the preset node priority strategy, the nodes in the original node feature sequence are reordered to obtain the target node feature sequence.

[0085] The global features are obtained based on the target feature node sequence.

[0086] Optionally, the node features can be flattened to obtain an original sorted sequence of node features, such as ABCDEF. Following a preset node priority strategy, the nodes in the original sequence are reordered to obtain an adjusted target node feature sequence, such as DFCAEB. Then, the global dependencies of the reordered target node feature sequence can be captured to obtain global features.

[0087] In one embodiment of this application, the step of reordering the nodes in the original node feature sequence according to a preset node priority strategy includes:

[0088] Calculate the node importance index for each node feature in the original node feature sequence;

[0089] The node features in the original node feature sequence are reordered according to the node importance index from low to high.

[0090] Optionally, the node features can be flattened to obtain the original sorted node feature sequence, for example, ABCDEF. Then, the importance index of each node is calculated sequentially. This importance index is the degree of each node, which can be understood as the number of times each node appears as an endpoint of an edge. Specifically, it can be calculated using the following formula:

[0091]

[0092] Where I is the indicator function and M is the number of node indices. After sorting, the node sequence is processed using the Mmaba module.

[0093] Based on this importance index, the original sorted node feature sequence is reordered according to the node importance index from low to high, that is, the high importance nodes are placed at the end of the sequence, to obtain the target node feature sequence, such as DFCAEB, in order to improve access to contextual information.

[0094] In one embodiment of this application, converting the MRI image to be detected and the PET image to be detected into MRI image structures and PET image structures, respectively, includes:

[0095] The MRI image and the PET image to be detected are segmented to obtain multiple image blocks, and the image blocks are used as nodes.

[0096] The edges between nodes are obtained based on spatial relationships;

[0097] The weight of each edge is determined based on the correlation between the endpoint nodes of each edge.

[0098] Based on the nodes, edges, and edge weights, the MRI image structure and PET image structure are obtained.

[0099] like Figure 3 As shown, the input MRI and PET images to be detected are converted from 3D images into graph structures. First, downsampling is performed using 3D convolution. Then, a block-based strategy is used to segment the downsampling image into non-overlapping 3D image blocks, which can be achieved using the following formula:

[0100]

[0101] The Unfold operation simulates a sliding window, reshaping a 3D image into a series of non-overlapping 3D image blocks. d p h p w These represent the depth, height, and width dimensions of each image patch. The above operations generate the feature tensor of the image patch. in This indicates the total number of image patches.

[0102] Features of each image patch Flattened into a node x i ∈R d , where d = C·p d ·p h ·pw This refers to the number of features. Before flattening, a 26-neighborhood connection model can be used to form edges between nodes based on spatial relationships. This is achieved by calculating the number of nodes paired with x. i and x j The correlation coefficients between the data, such as the Pearson correlation coefficient, are used to determine the marginal weights, thereby obtaining graph structure data. Graph structures can effectively represent the topological information of the brain.

[0103] In one embodiment of this application, the feature fusion of the image features and text features to obtain fused features includes:

[0104] The image features are linearly transformed to obtain multiple calculation factors;

[0105] Attention calculations and linear transformations are performed on the text features and multiple computational factors to obtain the fused features.

[0106] Optionally, such as Figure 4 As shown, the adaptive fusion module can include a traditional transformer layer and a multilayer perceptron (MLP). The traditional transformer layer can include an attention layer and a feedforward layer. Text features are processed through a traditional transformer layer, which consists of an attention mechanism and a feedforward layer, while image representations are transformed into multiple computational factors through linear layers. These computational factors participate in the computation of the attention mechanism and the MLP, respectively, thereby achieving the injection of cross-modal information.

[0107] The calculation factors may include three, namely: in, As a weighting factor, it is used to scale or weight text features so that image information can influence the transformation process of text features; This is a bias factor used to offset text features and adjust the reference point for the transformation. It is a gating factor used to control the intensity of information flow through multiplication operations. When it is close to 0, information transmission is suppressed, and when it is close to 1, information is allowed to pass through.

[0108] In one embodiment of this application, the step of performing attention calculation and linear transformation on the text features and multiple calculation factors to obtain the fused features includes:

[0109] The text features are subjected to layer normalization to obtain the layer-normalized text features.

[0110] An attention mechanism is used to calculate the text features after the layer normalization process and multiple calculation factors to obtain preliminary fused features.

[0111] The preliminary fused features are input into a multilayer perceptron for processing to obtain the final fused features.

[0112] Optionally, the initial feature representation of the text Layer normalization is performed to stabilize the numerical distribution and facilitate computation. Then, the values ​​obtained through the first MLP transformation are... A linear transformation is performed on the input table features, and then the transformed table features are passed through an attention layer. It is used as a gating parameter in the calculation and is combined with the initial feature representation of the text. By adding image-related information into the text feature representation using an attention mechanism, an updated feature representation is obtained. Then, to Perform layer normalization again, and obtain it through a second MLP transformation. Perform a linear transformation, then a nonlinear transformation using a multilayer perceptron (MLP), and then... As a gating parameter, image information is incorporated into the tabular features via an MLP (Multi-Level Processing) approach to obtain a further updated feature representation.

[0113] Finally, the hybrid representation learned after cross-modal information injection will be... Inputting the data into the classification header yields the classification results, which may include SMCI (Stability MCI) and PMCI (Progressive MCI).

[0114] It should be noted that the above text features are... Figure 4 The table feature in the text, because the text is organized in a tabular format, therefore... Figure 4 The table is used to represent the data.

[0115] Among them, three calculation factors They are represented by the following formulas respectively:

[0116]

[0117] in, As a weighting factor, it is used to scale or weight text features so that image information can influence the transformation process of text features; This is a bias factor used to offset text features and adjust the reference point for the transformation. It is a gating factor used to control the intensity of information flow through multiplication operations. When it is close to 0, information transmission is suppressed, and when it is close to 1, information is allowed to pass through.

[0118] Combining the above three computational factors into the attention mechanism and the multilayer perceptron computation for cross-modal information injection can be expressed as:

[0119]

[0120] in, as well as Can be Figure 4 The weighting factor, bias factor, and gating factor obtained after the first MLP processing are... as well as for Figure 4 The weight factor, bias factor, and gating factor are obtained after the second MLP processing. LayerNorm is the layer normalization, and Attention is the attention layer.

[0121] It should be noted that this application effectively combines image modality and text modality. To further verify the effectiveness of this model, it is compared with a model that uses only a single modality.

[0122] To accurately track the progression of MCI over time, cohorts were carefully selected from the ADNI and OASIS-3 datasets. Baseline patients with MCI were identified according to the Petersen criteria using established neuroimaging protocols, and their diagnostic outcomes were monitored through a series of clinical assessments over more than 24 months. Patients who progressed to AD within 36 months were classified as progressive MCI (pMCI=1), while those who retained their MCI diagnosis after this period were defined as stable MCI (sMCI=0).

[0123] The final cohort consisted of 305 samples from ADNI (244 for training and 61 for testing) and 140 samples from OASIS-3 (111 for training and 29 for testing). Each case included comprehensive CT imaging, demographic and clinical information including sex, age, MRI, and duration of illness. Each case was categorized as having a good or poor prognosis.

[0124] In terms of data preprocessing, demographic and clinical information is converted into text-based medical reports for each patient.

[0125] In the experiment, the following comparative experiments were used to evaluate the performance of the model: (1) pure text model: classification was performed using only demographic and clinical medical information; (2) pure vision model: classification was performed using only MRI image data and generated PET images. The results obtained by the above methods were compared with the results obtained using the method of this application. The Adam optimizer was used in the experiment. The training period, learning rate, and batch size were set to 40, 0.0001, and 2, respectively. The experimental results are shown in Table 1 below:

[0126] Table 1 shows the experimental results of this model and the model using only a single modality in terms of accuracy, recall, precision, F1 score, and AUC:

[0127]

[0128] Both plain text and pure vision models directly input features extracted from their respective encoders into multi-head self-attention blocks, which are then fed into the classification module. As shown in Table 1 above, compared to the plain text model, the method in this application demonstrates a significant improvement, exceeding it by 5.92% in accuracy and 0.0549 in AUC. The method in this application learns modal fusion features incorporating richer demographic and clinical information. This enhances the extraction of more contextual information, thereby facilitating more accurate prognostic predictions. Therefore, this application achieves significant improvements in accuracy, recall, precision, F1 score, and AUC.

[0129] In the embodiments of this application, generating PET images based on MRI images to be detected can compensate for the impact of missing modality information. Converting the MRI and PET images to be detected into graph structures allows for a more intuitive presentation of the complex connections between brain regions. The converted graph structure data can be seamlessly integrated with graph neural networks, providing them with an input format more consistent with the topological characteristics of the brain network. This helps the model learn brain network topology information more efficiently, uncover potential pathological features, and provide strong support for the early diagnosis of neurodegenerative diseases such as Alzheimer's disease. Furthermore, in the modality fusion stage, deep fusion of image and text features is achieved by adjusting the attention mechanism. This integrates complementary information from multiple sources, enhances model robustness, and uncovers potential correlations between modalities, thereby improving classification accuracy and highly meeting the practical needs of comprehensive clinical diagnosis.

[0130] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0131] In one embodiment, a graph neural network-based classification device is provided, which corresponds one-to-one with the graph neural network-based classification methods described in the above embodiments. For example... Figure 5 As shown, the graph neural network-based classification device includes a target image acquisition unit 10, a graph construction unit 20, an image feature acquisition unit 30, a text feature acquisition unit 40, and a feature fusion and classification unit 50. Detailed descriptions of each functional module are as follows:

[0132] The image acquisition unit 10 is used to acquire the MRI image to be detected and the text information to be detected corresponding to the MRI image to be detected, and to generate the PET image to be detected based on the MRI image to be detected;

[0133] Graph construction unit 20 is used to convert the MRI image to be detected and the PET image to be detected into MRI graph structure and PET graph structure, respectively, wherein the MRI graph structure and the PET graph structure include node features and edge features, respectively.

[0134] The image feature acquisition unit 30 is used to obtain image features based on the node features and edge features;

[0135] The text feature acquisition unit 40 is used to obtain text features based on the text information to be detected;

[0136] The feature fusion and classification unit 50 is used to fuse the image features and text features to obtain fused features, and to obtain a classification result based on the fused features.

[0137] In one embodiment of this application, the image feature acquisition unit 30 is further configured to:

[0138] The node features are input into the graph neural network module for processing to obtain global features;

[0139] Based on the node features and edge features, the data are input into a message passing network for processing to obtain local features;

[0140] The image features are obtained based on the global and local features.

[0141] In one embodiment of this application, the image feature acquisition unit 30 is further configured to:

[0142] The node features are flattened to obtain the original node feature sequence;

[0143] According to the preset node priority strategy, the nodes in the original node feature sequence are reordered to obtain the target node feature sequence.

[0144] The global features are obtained based on the target feature node sequence.

[0145] In one embodiment of this application, the image feature acquisition unit 30 is further configured to:

[0146] Calculate the node importance index for each node feature in the original node feature sequence;

[0147] The node features in the original node feature sequence are reordered according to the node importance index from low to high.

[0148] In one embodiment of this application, the image acquisition unit 10 to be detected is further configured to:

[0149] The MRI image and the PET image to be detected are segmented to obtain multiple image blocks, and the image blocks are used as nodes.

[0150] The edges between nodes are obtained based on spatial relationships;

[0151] The weight of each edge is determined based on the correlation between the endpoint nodes of each edge.

[0152] Based on the nodes, edges, and edge weights, the MRI image structure and PET image structure are obtained.

[0153] In one embodiment of this application, the feature fusion and classification unit 50 is further configured to:

[0154] The image features are linearly transformed to obtain multiple calculation factors;

[0155] Attention calculations and linear transformations are performed on the text features and multiple computational factors to obtain the fused features.

[0156] In one embodiment of this application, the feature fusion and classification unit 50 is further configured to:

[0157] The text features are subjected to layer normalization to obtain the layer-normalized text features.

[0158] An attention mechanism is used to calculate the text features after the layer normalization process and multiple calculation factors to obtain preliminary fused features.

[0159] The preliminary fused features are input into a multilayer perceptron for processing to obtain the final fused features.

[0160] In the embodiments of this application, generating PET images based on MRI images to be detected can compensate for the impact of missing modality information. Converting the MRI and PET images to be detected into graph structures allows for a more intuitive presentation of the complex connections between brain regions. The converted graph structure data can be seamlessly integrated with graph neural networks, providing them with an input format more consistent with the topological characteristics of the brain network. This helps the model learn brain network topology information more efficiently, uncover potential pathological features, and provide strong support for the early diagnosis of neurodegenerative diseases such as Alzheimer's disease. Furthermore, in the modality fusion stage, deep fusion of image and text features is achieved by adjusting the attention mechanism. This integrates complementary information from multiple sources, enhances model robustness, and uncovers potential correlations between modalities, thereby improving classification accuracy and highly meeting the practical needs of comprehensive clinical diagnosis.

[0161] For specific limitations regarding the graph neural network-based classification device, please refer to the limitations of the graph neural network-based classification method above, which will not be repeated here. Each module in the aforementioned graph neural network-based classification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0162] In one embodiment, a computer device is provided, which may be a terminal device, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a readable storage medium storing computer-readable instructions. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer-readable instructions implement a classification method based on a graph neural network. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0163] In this embodiment of the application, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, it implements the steps of the classification method based on graph neural networks described above.

[0164] In one embodiment of the application, a readable storage medium is provided, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, they implement the steps of the classification method based on graph neural networks described above.

[0165] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0166] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0167] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A classification method based on graph neural networks, characterized in that, The method includes: Acquire the MRI image to be detected and the corresponding text information to be detected, and generate the PET image to be detected based on the MRI image to be detected; The MRI image and PET image to be detected are converted into MRI image structure and PET image structure, respectively, wherein the MRI image structure and PET image structure include node features and edge features, respectively. Based on the node features and edge features, image features are obtained, specifically including: inputting the node features into a graph neural network module for processing to obtain global features, inputting the node features and edge features into a message passing network for processing to obtain local features, and fusing the global features and local features to obtain image features; The text information to be detected is subjected to standardized encoding processing to obtain text features. The standardized encoding processing includes one-hot encoding of categorical features and normalization of continuous features. The image features are linearly transformed to obtain three calculation factors: weight factor, bias factor, and gating factor. After the text features are layer normalized, attention mechanism calculation and multilayer perceptron processing are performed sequentially in combination with the calculation factors to achieve adaptive cross-modal fusion of image features and text features to obtain fused features. Based on the fused features, a classification result is obtained, which includes stable mild cognitive impairment or progressive mild cognitive impairment.

2. The classification method based on graph neural networks as described in claim 1, characterized in that, The step of inputting the node features into the graph neural network module for processing to obtain global features includes: The node features are flattened to obtain the original node feature sequence; According to the preset node priority strategy, the nodes in the original node feature sequence are reordered to obtain the target node feature sequence. The global features are obtained based on the target feature node sequence.

3. The classification method based on graph neural networks as described in claim 2, characterized in that, The step of reordering the nodes in the original node feature sequence according to a preset node priority strategy includes: Calculate the node importance index for each node feature in the original node feature sequence; The node features in the original node feature sequence are reordered according to the node importance index from low to high.

4. The classification method based on graph neural networks as described in any one of claims 1-3, characterized in that, The step of converting the MRI image and PET image to be detected into MRI image structure and PET image structure, respectively, includes: The MRI image and the PET image to be detected are segmented to obtain multiple image blocks, and the image blocks are used as nodes. The edges between nodes are obtained based on spatial relationships; The weight of each edge is determined based on the correlation between the endpoint nodes of each edge. Based on the nodes, edges, and edge weights, the MRI image structure and PET image structure are obtained.

5. The classification method based on graph neural networks as described in claim 1, characterized in that, After performing layer normalization on the text features, attention mechanism calculation and multilayer perceptron processing are performed sequentially in conjunction with the calculation factor to achieve adaptive cross-modal fusion of image features and text features to obtain fused features, including: The text features are subjected to layer normalization to obtain the layer-normalized text features. An attention mechanism is used to calculate the text features after the layer normalization process and multiple calculation factors to obtain preliminary fused features. The preliminary fused features are input into a multilayer perceptron for processing to obtain the final fused features.

6. A classification device based on a graph neural network, characterized in that, The device includes: The image acquisition unit is used to acquire the MRI image to be detected and the text information to be detected corresponding to the MRI image to be detected, and to generate the PET image to be detected based on the MRI image to be detected; The graph construction unit is used to convert the MRI image to be detected and the PET image to be detected into MRI graph structure and PET graph structure, respectively, wherein the MRI graph structure and the PET graph structure include node features and edge features, respectively. The image feature acquisition unit is used to obtain image features based on the node features and edge features, specifically including: inputting the node features into the graph neural network module for processing to obtain global features, inputting the node features and the edge features into the message passing network for processing to obtain local features, and fusing the global features and local features to obtain image features; The text feature acquisition unit is used to perform standardized encoding processing on the text information to be detected to obtain text features. The standardized encoding processing includes one-hot encoding of categorical features and normalization of continuous features. The feature fusion and classification unit is used to perform linear transformation on the image features to obtain three calculation factors: weight factor, bias factor, and gating factor. After performing layer normalization on the text features, attention mechanism calculation and multilayer perceptron processing are performed in sequence in combination with the calculation factors to achieve adaptive cross-modal fusion of image features and text features to obtain fused features. Based on the fused features, a classification result is obtained, which includes stable mild cognitive impairment or progressive mild cognitive impairment.

7. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the steps of the graph neural network-based classification method as described in any one of claims 1 to 5.

8. A readable storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by a processor, they implement the steps of the graph neural network-based classification method as described in any one of claims 1 to 5.

Images