Classification method and device based on graph neural network, computer equipment and medium

Abstract

The invention discloses a classification method, device and equipment based on a graph neural network and a storage medium, and the method comprises the steps: obtaining a to-be-detected MRI image and to-be-detected text information corresponding to the to-be-detected MRI image, and generating a to-be-detected PET image based on the to-be-detected MRI image; respectively converting the to-be-detected MRI image and the to-be-detected PET image into an MRI graph structure and a PET graph structure, wherein the MRI graph structure and the PET graph structure respectively comprise node features and edge features; obtaining image features based on the node features and the edge features; obtaining text features based on the to-be-detected text information; and carrying out feature fusion on the image features and the text features, and obtaining a classification result based on the fused features. Through cross-modal fusion of the text modality and the image modality, the influence of modal information loss can be made up, potential pathological features can be mined, multi-source complementary information can be integrated, the model robustness can be enhanced, and powerful support can be provided for early diagnosis of neurodegenerative diseases such as Alzheimer's disease.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a classification method, apparatus, computer equipment and medium based on graph neural network. Background Art

[0002] Alzheimer's disease (AD) is a progressive neurodegenerative disease that usually presents with mild cognitive impairment (MCI) in the early stages. Its irreversible pathological process and lack of effective treatment make early and accurate diagnosis the key to delaying the progression of the disease.

[0003] At present, clinical diagnosis mainly relies on multimodal imaging data such as magnetic resonance imaging (MRI) and positron emission tomography (PET), combined with comprehensive evaluation of patients' clinical indicators (such as age, cognitive scores, etc.). However, this process requires manual prediction by neurosurgeons, which consumes a lot of manpower and time, and the results are easily affected by the doctor's experience and subjective factors. With the advancement of science and technology, although machine learning technology has been initially applied to the early diagnosis of AD and has achieved certain results, it still has significant limitations. When processing multimodal data, existing methods generally lack deep fusion of cross-modal information, resulting in the failure to fully utilize the complementarity between data; at the same time, facing the complex topological structure of the brain, traditional feature extraction algorithms find it difficult to capture its subtle pathological changes, resulting in the loss of key diagnostic information. These technical bottlenecks directly restrict the accuracy and efficiency of early diagnosis of AD, and there is an urgent need to explore more efficient and objective intelligent diagnostic solutions. Summary of the Invention

[0004] Based on this, it is necessary to provide a classification method, device, computer equipment and medium based on graph neural network to address the above technical problems, so as to solve at least one problem existing in the above-mentioned prior art.

[0005] In the first aspect, a classification method based on a graph neural network is provided, comprising:

[0006] Acquire an MRI image to be detected and text information to be detected corresponding to the MRI image to be detected, and generate a PET image to be detected based on the MRI image to be detected;

[0007] Converting the to-be-detected MRI image and the to-be-detected PET image into an MRI graph structure and a PET graph structure, respectively, wherein the MRI graph structure and the PET graph structure include node features and edge features, respectively;

[0008] Obtaining image features based on the node features and edge features;

[0009] Obtaining text features based on the text information to be detected;

[0010] The image features and the text features are fused to obtain fused features, and a classification result is obtained based on the fused features.

[0011] In a possible implementation, obtaining image features based on the node features and edge features includes:

[0012] Input the node features into the graph neural network module for processing to obtain global features;

[0013] Input the node features and the edge features into a message passing network for processing to obtain local features;

[0014] The image features are obtained based on the global features and the local features.

[0015] In one possible implementation, inputting the node features into a graph neural network module for processing to obtain global features includes:

[0016] Flattening the node features to obtain an original node feature sequence;

[0017] According to a preset node priority strategy, each node in the original node feature sequence is reordered to obtain a target node feature sequence;

[0018] The global feature is obtained based on the target feature node sequence.

[0019] In a possible implementation, reordering the nodes in the original node feature sequence according to a preset node priority strategy includes:

[0020] Calculating the node importance index of each node feature in the original node feature sequence respectively;

[0021] The node features in the original node feature sequence are reordered according to the order of the node importance index from low to high.

[0022] In a possible implementation, converting the to-be-detected MRI image and the to-be-detected PET image into an MRI image structure and a PET image structure, respectively, includes:

[0023] Performing image segmentation on the MRI image to be detected and the PET image to be detected respectively to obtain a plurality of image blocks, and using the image blocks as nodes;

[0024] Get the edges between nodes based on spatial relationships;

[0025] Determine the weight of the corresponding edge based on the correlation between the endpoint nodes of each edge;

[0026] Based on the nodes, edges, and edge weights, the MRI graph structure and the PET graph structure are obtained.

[0027] In a possible implementation, the performing feature fusion on the image features and the text features to obtain fused features includes:

[0028] Performing a linear transformation on the image features to obtain a plurality of calculation factors;

[0029] Attention calculation and linear transformation are performed on the text feature and the plurality of calculation factors to obtain the fusion feature.

[0030] In a possible implementation, performing attention calculation and linear transformation on the text feature and the plurality of calculation factors to obtain the fusion feature includes:

[0031] Performing layer normalization processing on the text features to obtain layer-normalized text features;

[0032] Performing attention mechanism calculation on the text features after the layer normalization processing and the plurality of calculation factors to obtain preliminary fusion features;

[0033] The preliminary fusion features are input into a multi-layer perceptron for processing to obtain the final fusion features.

[0034] In a second aspect, a classification device based on a graph neural network is provided, comprising:

[0035] an image acquisition unit to be detected, configured to acquire an MRI image to be detected and text information to be detected corresponding to the MRI image to be detected, and generate a PET image to be detected based on the MRI image to be detected;

[0036] A graph construction unit, configured to convert the MRI image to be detected and the PET image to be detected into an MRI graph structure and a PET graph structure, respectively, wherein the MRI graph structure and the PET graph structure include node features and edge features, respectively;

[0037] An image feature acquisition unit, configured to obtain image features based on the node features and edge features;

[0038] A text feature acquisition unit, configured to obtain text features based on the text information to be detected;

[0039] The feature fusion and classification unit is used to perform feature fusion on the image features and text features to obtain fusion features, and to obtain classification results based on the fusion features.

[0040] In a third aspect, a computer device is provided, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein when the processor executes the computer-readable instructions, the steps of the graph neural network-based classification method as described above are implemented.

[0041] In a fourth aspect, a readable storage medium is provided, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the steps of the classification method based on the graph neural network as described above are implemented.

[0042] The above-mentioned classification method, device, computer equipment and medium based on graph neural network, and its method implementation include: obtaining an MRI image to be detected and text information to be detected corresponding to the MRI image to be detected, and generating a PET image to be detected based on the MRI image to be detected; converting the MRI image to be detected and the PET image to be detected into an MRI graph structure and a PET graph structure respectively, wherein the MRI graph structure and the PET graph structure respectively include node features and edge features; obtaining image features based on the node features and edge features; obtaining text features based on the text information to be detected; performing feature fusion on the image features and text features to obtain fusion features, and obtaining classification results based on the fusion features. In an embodiment of the present application, generating a PET image to be detected based on the MRI image to be detected can make up for the impact of the lack of modal information. Converting the MRI image to be detected and the PET image to be detected into a graph structure allows the complex connection relationship between brain regions to be intuitively presented. The converted graph structure data can be seamlessly integrated with graph neural networks, providing them with an input format more consistent with the topological characteristics of brain networks. This helps the model learn brain network topology information more efficiently, uncover potential pathological features, and provide strong support for the early diagnosis of neurodegenerative diseases such as Alzheimer's disease. Furthermore, during the modal fusion stage, by adjusting the attention mechanism to achieve a deep fusion of image and text features, it can not only integrate complementary information from multiple sources and enhance model robustness, but also uncover potential correlations between modalities, thereby improving classification accuracy. This is highly consistent with the actual needs of clinical multi-information comprehensive diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1This is a schematic diagram of an overall model architecture of a classification method based on a graph neural network in one embodiment of the present application;

[0045] Figure 2 This is a flow chart of a classification method based on a graph neural network in one embodiment of the present application;

[0046] Figure 3 This is a schematic diagram of a module structure of the Mamba module in one embodiment of the present application;

[0047] Figure 4 1 is a schematic diagram of a module structure of a modality fusion module in one embodiment of the present application;

[0048] Figure 5 This is a structural diagram of a classification device based on a graph neural network in one embodiment of the present application;

[0049] Figure 6 Schematic diagram of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0051] The classification method based on graph neural network provided in this embodiment can be applied to Figure 1In the model architecture, the model architecture may include a generative model module, a graph Mmaba architecture and an adaptive fusion module. First, the acquired MRI image to be detected is input into the generative model module, and in the generative model module, a PET image to be detected is generated based on the MRI image to be detected by a 3D generative adversarial network to enhance the details and biomarker information in the image modality and enrich the image details. Then, the MRI image to be detected and the PET image to be detected are input into the graph Mmaba architecture, which includes an Mmaba module and a message passing network. The MRI image to be detected and the PET image to be detected are respectively constructed into a graph structure, and the points and edges of the graph structure are embedded to obtain point (node) feature embedding and edge feature embedding. The point feature embedding is processed by the graph Mmaba module to obtain global features, and the point embedding features and edge embedding features are processed by the message passing network to obtain local features. The global features and local features are then fused through a multi-layer perceptron MLP to obtain image feature representation. Image representation and topological information can be learned from the image module. In order to better fuse image representation and text information, the text information can be organized into tabular information and the tabular feature representation can be extracted. Then, the image feature representation and the tabular feature representation extracted from the tabular data are fused through the adaptive modal fusion module. Finally, the fused mixed modal representation is input into the final classification head to obtain the classification result.

[0052] The tabular feature representation is obtained by organizing the text information corresponding to the MRI image to be tested into a table, for example, and then extracting features using a feature extraction model. This may include a table of clinical information of the user to be tested, including information such as gender, age, weight, symptoms, and disease stage.

[0053] In one embodiment, if Figure 2 As shown in the figure, a classification method based on graph neural network is provided, which includes the following steps:

[0054] In step S110, an MRI image to be detected and text information to be detected corresponding to the MRI image to be detected are acquired, and a PET image to be detected is generated based on the MRI image to be detected;

[0055] The MRI images to be tested are magnetic resonance imaging (MRI), which uses signals generated by the magnetic resonance phenomenon of hydrogen protons in the human body when excited by radio frequency pulses in a magnetic field. These signals are then processed by computers to reconstruct images of the body's internal structures. These images clearly display the brain's gray matter, white matter, ventricles, and other structures. They are highly sensitive to changes in brain morphology and structure, helping doctors detect brain abnormalities associated with Alzheimer's disease, such as hippocampal atrophy and white matter lesions.

[0056] PET images, which refer to positron emission tomography (PET), are obtained by injecting radioactive tracers into the human body. These tracers participate in metabolic processes within the body, and the decay of positrons produces annihilation radiation with electrons. The resulting gamma rays are detected by detectors, generating images that reflect metabolic activity within the body. In the diagnosis of Alzheimer's disease, the tracers commonly used in PET images can specifically bind to certain lesions in the brain, such as beta-amyloid plaques. This allows for intuitive visualization of metabolic changes in specific brain regions, helping to detect abnormalities in brain function early in the disease. Even before significant structural changes in the brain occur, areas of decreased metabolism can be detected, providing important evidence for early diagnosis.

[0057] The text information to be tested refers to the clinical information of the user to be tested, which may include text information such as gender, age, weight, symptoms, disease stage, etc. It can be obtained through doctor diagnosis, patient input, etc.

[0058] Specifically, the MRI image to be tested can be input into a generative model module, which is pre-installed with a 3D generative adversarial network. This 3D generative adversarial network can be composed of a generator and a discriminator. The generator structure adopts the traditional Unet structure, which includes two downsampling layers and two upsampling layers, with the middle layer being the bottleneck layer of the Vit model. The discriminator structure is the same as that of the discriminator in the PatchGan model. The specific process of generating the PET image to be tested based on the MRI image to be tested can be as follows: first, the input MRI sample image is subjected to preprocessing operations such as normalization and standardization. The PET sample image corresponding to the MRI sample image is also subjected to the same preprocessing operations. Then, based on the generator, the preprocessed MRI sample image is converted into a feature representation similar to the PET image. The discriminator is used to determine the difference between the generated similar PET image and the real PET image. Based on the output of the discriminator, the loss function of the generator and the discriminator is calculated. For the generator, its loss function aims to make the generated image as close to the real PET image as possible to deceive the discriminator. The 3D generative adversarial network undergoes multiple rounds of iterative training. During training, the generator continuously improves the quality of generated images, while the discriminator continuously improves its ability to discern image authenticity. The two work together, both competing and collaborating, to make the generated PET images increasingly similar to real PET images. Once the pre-defined convergence criteria are met, such as when the number of iterations reaches a certain value or when the loss of the generator and discriminator falls within a specific range, the MRI image to be tested is fed into the generator to generate the final PET image to be tested.

[0059] In step S120, the MRI image to be detected and the PET image to be detected are converted into an MRI graph structure and a PET graph structure respectively, wherein the MRI graph structure and the PET graph structure respectively include node features and edge features;

[0060] Optionally, the MRI image to be detected and the PET image to be detected are respectively input into the Mmaba architecture for processing, such as Figure 3 As shown, the graph neural network architecture (graph Mmaba architecture) may include part a and part b, part a is used to convert the MRI image to be detected and the PET image to be detected into an MRI graph structure and a PET graph structure respectively, and part b is used to obtain image features based on the MRI graph structure and the PET graph structure. Since the original images of the MRI image to be detected and the PET image to be detected are large, 3D CNN (convolution) can be used to downsample the image. Then, the downsampled image is divided into non-overlapping image blocks through a blocking strategy, and the image blocks are used as nodes. The 26-neighborhood connection model is used to form edges between nodes based on spatial relationships, and the edge weights are determined by calculating the Pearson correlation coefficient between the nodes, thereby obtaining graph structure data. The graph structure can effectively represent the topological information of the brain.

[0061] In step S130, image features are obtained based on the node features and edge features;

[0062] like Figure 1 As shown, node features and edge features can be embedded separately to obtain node embedded features and edge embedded features, thereby converting node features and edge features into feature representations suitable for model processing. The point embedded features can then be input into the Mmaba module for processing to obtain global features. Simultaneously, the point embedded features and edge embedded features are input into a message passing network for processing to obtain local features. Finally, the global features output by the Mmaba module and the local features output by the message passing network are input into a multi-layer perceptron (MLP) for fusion to obtain the image feature representation.

[0063] In step S140, text features are obtained based on the text information to be detected;

[0064] Optionally, clinical information of the user to be tested can be collected and organized into standardized tabular data based on clinical attributes, such as gender, age, weight, symptom type, disease stage, etc. Each row represents a clinical information of the user to be tested. The text category can then be converted into a numerical representation. For example, a binary variable is used to encode gender, such as male is encoded as 1 and female is encoded as 0. For multi-classification variables, one-hot encoding (One-Hot Encoding) or word embedding (Word Embedding) can be used for encoding. For disease stages, such as Alzheimer's disease stages, it can be mapped to ordered values ​​(mild → 1, moderate → 2, severe → 3), and continuous values ​​(such as age, weight, cognitive score) are normalized or standardized. All processed fields are combined into a two-dimensional matrix, with each column corresponding to a feature, to obtain the text feature representation, that is, the tabular feature representation.

[0065] In step S150 , the image features and the text features are fused to obtain fused features, and a classification result is obtained based on the fused features.

[0066] like Figure 4 As shown, the adaptive fusion module may include a traditional transformer layer and a multi-layer perceptron MLP. The traditional transformer layer may include an attention layer and a feedforward layer. First, the image feature representation F I It can be converted into three calculation factors through the multi-layer perceptron MLP, which are MLP includes multiple linear layers for feature transformation, mapping image features to a suitable space to prepare for subsequent cross-modal information injection. Perform layer normalization (LayerNorm) to stabilize the value distribution and facilitate calculation. Perform a linear transformation on the input table features, and then pass the linearly transformed table features through the attention layer or feedforward layer to As a gate parameter, it is calculated and combined with the initial feature representation of the text Add, integrate the image related information into the text feature representation in the form of attention mechanism, and obtain the updated feature representation Then, yes Perform layer normalization again and pass Perform linear transformation, and then perform nonlinear transformation through multi-layer perceptron MLP. Participate in the calculation as a gate parameter, integrate the image information into the table features through MLP, and obtain a further updated feature representation

[0067] Finally, the hybrid representation learned after cross-modal information injection is The input is fed into the classification head, i.e., a linear layer, to obtain the classification results, which may include SMCI (stable mild cognitive impairment) and PMCI (progressive mild cognitive impairment). The classification process can be specifically represented by the following formula:

[0068]

[0069] It should be noted that the above text features are Figure 4 The table features in the text are organized in table form. Figure 4 It is represented by table features.

[0070] in, is a weight factor used to scale or weight text features so that image information can affect the transformation process of text features; is the bias factor, which is used to offset the text features and adjust the base point of the transformation; It is a gating factor used to control the flow intensity of information through multiplication operation. When it is close to 0, information transmission is inhibited, and when it is close to 1, information is allowed to pass.

[0071] In an embodiment of the present application, a PET image to be detected is generated based on the MRI image to be detected, which can make up for the impact of the lack of modal information. The MRI image to be detected and the PET image to be detected are converted into a graph structure, so that the complex connection relationship between brain regions can be intuitively presented. The converted graph structure data can be seamlessly connected with the graph neural network, providing the graph neural network with an input form that is more in line with the topological characteristics of the brain network, thereby helping the model to learn the topological information of the brain network more efficiently, explore potential pathological characteristics, and provide strong support for the early diagnosis of neurodegenerative diseases such as Alzheimer's disease. In addition, in the modal fusion stage, the deep fusion of image and text features is achieved by adjusting the attention mechanism, which can not only integrate multi-source complementary information and enhance the robustness of the model, but also explore the potential correlation between modalities, thereby improving the classification accuracy, which is highly consistent with the actual needs of clinical multi-information comprehensive diagnosis.

[0072] In one embodiment of the present application, obtaining image features based on the node features and edge features includes:

[0073] Input the node features into the graph neural network module for processing to obtain global features;

[0074] Input the node features and the edge features into a message passing network for processing to obtain local features;

[0075] The image features are obtained based on the global features and the local features.

[0076] Optionally, a graph Mamba architecture may be used to generate image features, which may include a message passing network and an Mmaba module. Figure 1 As shown in Figure 2, the node features can be embedded (Embedding) to convert the node features into a feature representation suitable for model processing. Thus, the node embedding feature ( Figure 1 The point embedding features are shown in Figure 2). The point embedding features can then be input into the Mmaba module for processing. The Mmaba module embeds a selective state space model (SSM). First, the node embedding representation is input into the convolutional projection layer (ConvProjection) and the gated projection layer (GateProjection), respectively. The convolutional projection layer performs convolution operations and projection transformations, and the gated projection layer calculates a gating signal, which is used to control the flow of data. The output of the ConvProjection is then input into the convolutional layer (Conv), which performs a convolution operation. The convolutional layer calculates local features by sliding the convolution kernel over the data. The output of the convolutional layer is nonlinearly transformed using an activation function such as ReLU. The SSM component then captures the dynamic changes and global dependencies of the feature data processed by the activation function over time or sequence. Finally, the features processed by the SSM component are cross-multiplied with the gated signal, and the cross-multiplication result is projected through the output projection layer (OutputProjection), mapping the features to the final output space to obtain the global features.

[0077] At the same time, edge features can be embedded (embedded) to obtain edge embedding features. These node embedding features and edge embedding features are then fed into a message passing network (MPNN), which focuses on capturing local neighborhood features. For example, for each node in the MPNN, messages can be computed using a learnable message function (typically a neural network) based on its own node embedding features, the edge embedding features of its connected edges, and the node embedding features of its neighboring nodes. Each node aggregates the messages received from all of its neighboring nodes. Using these aggregated messages, an update function (such as a gated recurrent unit (GRU) or a long short-term memory (LSTM) network) is used to update the node's state. Through multiple iterations, the scope of local neighborhood feature capture can be continuously expanded, ultimately enabling the node to effectively capture a rich set of local neighborhood features.

[0078] The above global feature and local feature extraction process can be expressed by the following formula:

[0079] X i+1i =MPNN(X i ,E i)+Mamba(X i ).

[0080] Among them, X i Represents node embedding representation, Mamba(X i ) represents the global feature, E i represents edge node embedding, MPNN(X i ,E i ) represents local features.

[0081] Finally, the global features and local features are input into the multi-layer perceptron (MLP) for linear transformation to obtain the final image feature representation.

[0082] In one embodiment of the present application, the node features are input into a graph neural network module for processing to obtain global features, including:

[0083] Flattening the node features to obtain an original node feature sequence;

[0084] According to a preset node priority strategy, each node in the original node feature sequence is reordered to obtain a target node feature sequence;

[0085] The global feature is obtained based on the target feature node sequence.

[0086] Optionally, the node features can be flattened to obtain an original ordered node feature sequence, such as ABCDEF. According to a preset node priority strategy, the nodes in the original node feature sequence are reordered to obtain an adjusted target node feature sequence, such as DFCAEB. The reordered target node feature sequence can then be used to capture global dependencies to obtain global features.

[0087] In one embodiment of the present application, reordering the nodes in the original node feature sequence according to a preset node priority strategy includes:

[0088] Calculating the node importance index of each node feature in the original node feature sequence respectively;

[0089] The node features in the original node feature sequence are reordered according to the order of the node importance index from low to high.

[0090] Optionally, the node features can be flattened to obtain the original sorted node feature sequence, for example, ABCDEF. Then, the importance index of each node is calculated in turn. This importance index is the degree of each node, which can be understood as the number of times each node appears as an edge endpoint. It can be calculated using the following formula:

[0091]

[0092] Where I is the indicator function and M is the number of node indexes. After sorting, the node sequence is processed using the Mmaba module.

[0093] Based on this importance index, the originally sorted node feature sequence is reordered in order of the node importance index from low to high, that is, high-importance nodes are placed at the end of the sequence, to obtain the target node feature sequence, such as DFCAEB, to improve access to context information.

[0094] In one embodiment of the present application, converting the to-be-detected MRI image and the to-be-detected PET image into an MRI image structure and a PET image structure, respectively, includes:

[0095] Performing image segmentation on the MRI image to be detected and the PET image to be detected respectively to obtain a plurality of image blocks, and using the image blocks as nodes;

[0096] Get the edges between nodes based on spatial relationships;

[0097] Determine the weight of the corresponding edge based on the correlation between the endpoint nodes of each edge;

[0098] Based on the nodes, edges, and edge weights, the MRI graph structure and the PET graph structure are obtained.

[0099] like Figure 3 As shown in the figure, the input MRI image and PET image to be detected are converted from three-dimensional images to graph structures. First, downsampling is performed through 3D convolution, and then the downsampled image is segmented into non-overlapping 3D image blocks through a block strategy. This can be achieved by the following formula:

[0100]

[0101] Among them, the Unfold operation simulates a sliding window and reshapes a 3D image into a series of non-overlapping 3D image blocks. d , p h , p w ) are the depth, height and width of each image block respectively. The above operations generate the feature tensor of the image block in Indicates the total number of image blocks.

[0102] Each image block features is flattened into a node x i ∈R d , where d = C·p d ·p h ·pw is the number of features. Before flattening, the 26-neighborhood connection model can be used to form edges between nodes based on spatial relationships. By calculating the node pair x i and x j The correlation coefficient between them, such as the Pearson correlation coefficient, is used to determine the edge weight, thereby obtaining graph structure data, which can effectively represent the topological information of the brain.

[0103] In one embodiment of the present application, the image features and text features are fused to obtain fused features, including:

[0104] Performing a linear transformation on the image features to obtain a plurality of calculation factors;

[0105] Attention calculation and linear transformation are performed on the text feature and the plurality of calculation factors to obtain the fusion feature.

[0106] Alternatively, as Figure 4 As shown, the adaptive fusion module can include traditional transformer layers and multi-layer perceptrons (MLPs). Traditional transformer layers can include attention layers and feedforward layers. Text features are passed through a traditional transformer layer, namely an attention mechanism and a feedforward layer, while image representations are converted into multiple computational factors through a linear layer. These computational factors participate in the calculation of the attention mechanism and the multi-layer perceptron, thereby achieving cross-modal information injection.

[0107] Among them, the calculation factors may include 3, namely in, is a weight factor used to scale or weight text features so that image information can affect the transformation process of text features; is the bias factor, which is used to offset the text features and adjust the base point of the transformation; It is a gating factor used to control the flow intensity of information through multiplication operation. When it is close to 0, information transmission is inhibited, and when it is close to 1, information is allowed to pass.

[0108] In one embodiment of the present application, performing attention calculation and linear transformation on the text feature and the plurality of calculation factors to obtain the fusion feature includes:

[0109] Performing layer normalization processing on the text features to obtain layer-normalized text features;

[0110] Performing attention mechanism calculation on the text features after the layer normalization processing and the plurality of calculation factors to obtain preliminary fusion features;

[0111] The preliminary fusion features are input into a multi-layer perceptron for processing to obtain the final fusion features.

[0112] Optionally, an initial feature representation of the text Perform layer normalization (LayerNorm) to stabilize the value distribution and facilitate calculation. Then, the Perform a linear transformation on the input table features, and then pass the linearly transformed table features through the attention layer to As a gate parameter, it is calculated and combined with the initial feature representation of the text Add, integrate the image related information into the text feature representation in the form of attention mechanism, and obtain the updated feature representation Then, yes The layer is normalized again and converted through the second MLP Perform linear transformation, then perform nonlinear transformation through multi-layer perceptron MLP, and Participate in the calculation as a gate parameter, integrate the image information into the table features through MLP, and obtain a further updated feature representation

[0113] Finally, the hybrid representation learned after cross-modal information injection is The data are input into the classification head to obtain the classification results, which may include SMCI (stable MCI) and PMCI (progressive MCI).

[0114] It should be noted that the above text features are Figure 4 The table features in the text are organized in table form. Figure 4 It is represented by table features.

[0115] Among them, the three calculation factors They are expressed by the following formulas:

[0116]

[0117] in, is a weight factor used to scale or weight text features so that image information can affect the transformation process of text features; is the bias factor, which is used to offset the text features and adjust the base point of the transformation; It is a gating factor used to control the flow intensity of information through multiplication operation. When it is close to 0, information transmission is inhibited, and when it is close to 1, information is allowed to pass.

[0118] Mixing the above three calculation factors into the calculation of the attention mechanism and the multi-layer perceptron to inject cross-modal information can be expressed as:

[0119]

[0120] in, as well as Can Figure 4 The weight factor, bias factor, and gating factor obtained after the first MLP processing in as well as for Figure 4 The weight factor, bias factor, and gating factor obtained after the second MLP processing, LayerNorm is layer normalization, and Attention is the attention layer.

[0121] It should be noted that this application effectively combines the image modality and the text modality. In order to further verify the effect of this model, this model is compared with a model that only uses a single modality.

[0122] To accurately track the progression of MCI over time, a cohort was carefully selected from the ADNI and OASIS-3 datasets. Using an established neuroimaging protocol, patients with baseline MCI were identified according to the Petersen criteria, and their diagnostic outcomes were monitored through serial clinical assessments over 24 months. Patients who progressed to AD within 36 months were classified as having progressive MCI (pMCI = 1), while those who retained the MCI diagnosis after this period were identified as having stable MCI (sMCI = 0).

[0123] The final cohort consisted of 305 samples from ADNI, divided into 244 for training and 61 for testing, and 140 samples from OASIS-3, divided into 111 for training and 29 for testing. Each case included comprehensive CT imaging, including demographic and clinical information, including: sex, age, MRI, and illness duration. Each case was labeled as having a good or poor prognosis.

[0124] In terms of data preprocessing, demographic and clinical information were converted into textual medical reports for each patient.

[0125] In the experiment, the following comparative experiments were used to evaluate the performance of the model: (1) a text-only model: classification was performed using only demographic and clinical information; (2) a visual-only model: classification was performed using only MRI imaging data and generated PET images. The results obtained using the above methods were compared with those obtained using the method of the present application. The experiment used the Adam optimizer. The training epochs, learning rate, and batch size were set to 40, 0.0001, and 2, respectively. The experimental results are shown in Table 1 below:

[0126] Table 1 shows the experimental results of this model and the model using only a single modality in terms of accuracy, recall, precision, F1 score, and AUC:

[0127]

[0128] The text-only and vision-only models directly input the features extracted from the corresponding encoders into the multi-head self-attention block and then into the classification module. As can be seen from Table 1 above, compared with the text-only model, the method of the present application shows significant improvement, exceeding its accuracy by 5.92% and its AUC indicator by 0.0549. The method of the present application learns modality fusion features that contain richer demographic and clinical information. This enhances the extraction of more contextual information, thereby promoting more accurate prognosis prediction. As a result, the present application has achieved significant improvements in the indicators of accuracy, recall, precision, F1 score, and AUC.

[0129] In an embodiment of the present application, a PET image to be detected is generated based on the MRI image to be detected, which can make up for the impact of the lack of modal information. The MRI image to be detected and the PET image to be detected are converted into a graph structure, so that the complex connection relationship between brain regions can be intuitively presented. The converted graph structure data can be seamlessly connected with the graph neural network, providing the graph neural network with an input form that is more in line with the topological characteristics of the brain network, thereby helping the model to learn the topological information of the brain network more efficiently, explore potential pathological characteristics, and provide strong support for the early diagnosis of neurodegenerative diseases such as Alzheimer's disease. In addition, in the modal fusion stage, the deep fusion of image and text features is achieved by adjusting the attention mechanism, which can not only integrate multi-source complementary information and enhance the robustness of the model, but also explore the potential correlation between modalities, thereby improving the classification accuracy, which is highly consistent with the actual needs of clinical multi-information comprehensive diagnosis.

[0130] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0131] In one embodiment, a classification device based on a graph neural network is provided, and the classification device based on a graph neural network corresponds one-to-one to the classification method based on a graph neural network in the above embodiment. Figure 5 As shown, the classification device based on graph neural network includes an image acquisition unit 10 to be detected, a graph construction unit 20, an image feature acquisition unit 30, a text feature acquisition unit 40 and a feature fusion and classification unit 50. The functional modules are described in detail as follows:

[0132] The image acquisition unit 10 is configured to acquire an MRI image to be detected and text information to be detected corresponding to the MRI image to be detected, and generate a PET image to be detected based on the MRI image to be detected;

[0133] A graph construction unit 20, configured to convert the MRI image to be detected and the PET image to be detected into an MRI graph structure and a PET graph structure, respectively, wherein the MRI graph structure and the PET graph structure include node features and edge features, respectively;

[0134] An image feature acquisition unit 30, configured to obtain image features based on the node features and edge features;

[0135] A text feature acquisition unit 40 is configured to obtain text features based on the text information to be detected;

[0136] The feature fusion and classification unit 50 is used to perform feature fusion on the image features and text features to obtain fused features, and obtain classification results based on the fused features.

[0137] In one embodiment of the present application, the image feature acquisition unit 30 is further configured to:

[0138] Input the node features into the graph neural network module for processing to obtain global features;

[0139] Input the node features and the edge features into a message passing network for processing to obtain local features;

[0140] The image features are obtained based on the global features and the local features.

[0141] In one embodiment of the present application, the image feature acquisition unit 30 is further configured to:

[0142] Flattening the node features to obtain an original node feature sequence;

[0143] According to a preset node priority strategy, each node in the original node feature sequence is reordered to obtain a target node feature sequence;

[0144] The global feature is obtained based on the target feature node sequence.

[0145] In one embodiment of the present application, the image feature acquisition unit 30 is further configured to:

[0146] Calculating the node importance index of each node feature in the original node feature sequence respectively;

[0147] The node features in the original node feature sequence are reordered according to the order of the node importance index from low to high.

[0148] In one embodiment of the present application, the to-be-detected image acquisition unit 10 is further configured to:

[0149] Performing image segmentation on the MRI image to be detected and the PET image to be detected respectively to obtain a plurality of image blocks, and using the image blocks as nodes;

[0150] Get the edges between nodes based on spatial relationships;

[0151] Determine the weight of the corresponding edge based on the correlation between the endpoint nodes of each edge;

[0152] Based on the nodes, edges, and edge weights, the MRI graph structure and the PET graph structure are obtained.

[0153] In one embodiment of the present application, the feature fusion and classification unit 50 is further configured to:

[0154] Performing a linear transformation on the image features to obtain a plurality of calculation factors;

[0155] Attention calculation and linear transformation are performed on the text feature and the plurality of calculation factors to obtain the fusion feature.

[0156] In one embodiment of the present application, the feature fusion and classification unit 50 is further configured to:

[0157] Performing layer normalization processing on the text features to obtain layer-normalized text features;

[0158] Performing attention mechanism calculation on the text features after the layer normalization processing and the plurality of calculation factors to obtain preliminary fusion features;

[0159] The preliminary fusion features are input into a multi-layer perceptron for processing to obtain the final fusion features.

[0160] In an embodiment of the present application, a PET image to be detected is generated based on the MRI image to be detected, which can make up for the impact of the lack of modal information. The MRI image to be detected and the PET image to be detected are converted into a graph structure, so that the complex connection relationship between brain regions can be intuitively presented. The converted graph structure data can be seamlessly connected with the graph neural network, providing the graph neural network with an input form that is more in line with the topological characteristics of the brain network, thereby helping the model to learn the topological information of the brain network more efficiently, explore potential pathological characteristics, and provide strong support for the early diagnosis of neurodegenerative diseases such as Alzheimer's disease. In addition, in the modal fusion stage, the deep fusion of image and text features is achieved by adjusting the attention mechanism, which can not only integrate multi-source complementary information and enhance the robustness of the model, but also explore the potential correlation between modalities, thereby improving the classification accuracy, which is highly consistent with the actual needs of clinical multi-information comprehensive diagnosis.

[0161] For the specific definition of the classification device based on graph neural network, please refer to the definition of the classification method based on graph neural network above, which will not be repeated here. Each module in the above-mentioned classification device based on graph neural network can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0162] In one embodiment, a computer device is provided. The computer device may be a terminal device, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium. The readable storage medium stores computer-readable instructions. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer-readable instructions are executed by the processor, a classification method based on a graph neural network is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0163] In an embodiment of the present application, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the steps of the above-mentioned graph neural network-based classification method are implemented.

[0164] In an embodiment of the application, a readable storage medium is provided, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the steps of the above-mentioned graph neural network-based classification method are implemented.

[0165] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they may include processes in the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0166] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0167] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A classification method based on graph neural network, characterized in that: The method comprises: Acquire an MRI image to be detected and text information to be detected corresponding to the MRI image to be detected, and generate a PET image to be detected based on the MRI image to be detected; Converting the to-be-detected MRI image and the to-be-detected PET image into an MRI graph structure and a PET graph structure, respectively, wherein the MRI graph structure and the PET graph structure include node features and edge features, respectively; Obtaining image features based on the node features and edge features; Obtaining text features based on the text information to be detected; The image features and the text features are fused to obtain fused features, and a classification result is obtained based on the fused features.

2. The classification method based on graph neural network according to claim 1, characterized in that The obtaining of image features based on the node features and edge features includes: Input the node features into the graph neural network module for processing to obtain global features; Input the node features and the edge features into a message passing network for processing to obtain local features; The image features are obtained based on the global features and the local features.

3. The classification method based on graph neural network according to claim 2, characterized in that: The node features are input into the graph neural network module for processing to obtain global features, including: Flattening the node features to obtain an original node feature sequence; According to a preset node priority strategy, each node in the original node feature sequence is reordered to obtain a target node feature sequence; The global feature is obtained based on the target feature node sequence.

4. The classification method based on graph neural network according to claim 3, characterized in that: The reordering of the nodes in the original node feature sequence according to the preset node priority strategy includes: Calculating the node importance index of each node feature in the original node feature sequence respectively; The node features in the original node feature sequence are reordered according to the order of the node importance index from low to high.

5. The graph neural network-based classification method according to any one of claims 1 to 4, characterized in that: The converting the to-be-detected MRI image and the to-be-detected PET image into an MRI image structure and a PET image structure respectively includes: Performing image segmentation on the MRI image to be detected and the PET image to be detected respectively to obtain a plurality of image blocks, and using the image blocks as nodes; Get the edges between nodes based on spatial relationships; Determine the weight of the corresponding edge based on the correlation between the endpoint nodes of each edge; Based on the nodes, edges, and edge weights, the MRI graph structure and the PET graph structure are obtained.

6. The graph neural network-based classification method according to any one of claims 1 to 4, characterized in that: The performing feature fusion on the image features and the text features to obtain fused features includes: Performing a linear transformation on the image features to obtain a plurality of calculation factors; Attention calculation and linear transformation are performed on the text feature and the plurality of calculation factors to obtain the fusion feature.

7. The classification method based on graph neural network according to claim 6, characterized in that: The performing attention calculation and linear transformation on the text feature and the plurality of calculation factors to obtain the fusion feature includes: Performing layer normalization processing on the text features to obtain layer-normalized text features; Performing attention mechanism calculation on the text features after the layer normalization processing and the plurality of calculation factors to obtain preliminary fusion features; The preliminary fusion features are input into a multi-layer perceptron for processing to obtain the final fusion features.

8. A classification device based on graph neural network, characterized in that: The device comprises: an image acquisition unit to be detected, configured to acquire an MRI image to be detected and text information to be detected corresponding to the MRI image to be detected, and generate a PET image to be detected based on the MRI image to be detected; A graph construction unit, configured to convert the MRI image to be detected and the PET image to be detected into an MRI graph structure and a PET graph structure, respectively, wherein the MRI graph structure and the PET graph structure include node features and edge features, respectively; An image feature acquisition unit, configured to obtain image features based on the node features and edge features; A text feature acquisition unit, configured to obtain text features based on the text information to be detected; The feature fusion and classification unit is used to perform feature fusion on the image features and text features to obtain fusion features, and to obtain classification results based on the fusion features.

9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein: When the processor executes the computer-readable instructions, the steps of the graph neural network-based classification method as described in any one of claims 1 to 7 are implemented.

10. A readable storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the graph neural network-based classification method as described in any one of claims 1 to 7 are implemented.

Images