A multi-modal medical data classification method, model, device and storage medium

Abstract

The present application relates to the technical field of medical data processing, in particular to a multi-modal medical data classification method, model, device and storage medium. The present application first encodes and aligns the original multi-modal medical data to obtain the multi-modal features after alignment, then converts the multi-modal features after alignment into graph data, then extracts the common features between the multi-modal and the unique features of each modality, and finally classifies the original multi-modal medical data according to the common features and the unique features. Since the common features reflect the correlation between the multi-modal data, and the unique features can reflect the specificity of the modality, the comprehensive use of the common features and the unique features can realize the accurate classification of the multi-modal medical data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical data processing, and particularly relates to a multi-modal medical data classification method, model, device and storage medium. BACKGROUND

[0002] Various medical examinations are performed on a patient, and thus multi-modal medical data of the same patient can be obtained, that is, one modality of medical data can be obtained through each medical examination. For example, medical data obtained through various examinations such as magnetic resonance imaging, computed tomography, ultrasonic examination and electrocardiogram examination constitutes multi-modal medical data. Through analysis and processing of the multi-modal medical data, classification of the multi-modal medical data can be achieved, and the classification result can be used to reflect the disease degree of the patient. The prior art only classifies the multi-modal medical data, and does not fully consider the identity and specificity between the multi-modal medical data, thereby reducing the accuracy of the classification result of the multi-modal medical data.

[0003] To sum up, the prior art reduces the accuracy of the classification result of the multi-modal medical data.

[0004] Therefore, the prior art still needs to be improved and enhanced. SUMMARY

[0005] To solve the above technical problems, the present application provides a multi-modal medical data classification method, model, device and storage medium, which solves the problem that the prior art reduces the accuracy of the classification result of the multi-modal medical data.

[0006] To achieve the above object, the present application adopts the following technical scheme:

[0007] In a first aspect, the present application provides a multi-modal medical data classification method, which comprises:

[0008] obtaining original multi-modal medical data, and performing encoding alignment processing on the original multi-modal medical data to obtain encoded and aligned multi-modal features;

[0009] converting the encoded and aligned multi-modal features into graph data, the graph data comprising nodes and edges between any two nodes, wherein each node is used to represent each modality, the feature of the node corresponds to the feature of the modality, and the feature of the edge corresponds to the features of the two nodes connected by the edge;

[0010] obtaining common features between the multi-modal and unique features of each modality according to the graph data;

[0011] obtaining a classification result of the original multi-modal medical data according to the common features and the unique features.

[0012] In an implementation manner, the encoding alignment processing on the original multi-modal medical data obtains an encoding alignment multi-modal feature, and the encoding alignment processing includes:

[0013] determining a type of each of the original multi-modal medical data, determining an encoder corresponding to each of the types, and extracting a feature of the multi-modal medical data through the encoder to obtain an encoding feature of the multi-modal medical data;

[0014] adjusting dimensions of the encoding features of the multi-modal medical data to the same dimension to obtain the encoding alignment multi-modal feature.

[0015] In an implementation manner, the encoding alignment multi-modal feature is converted into graph data, and the conversion includes:

[0016] each of the modalities corresponds to a node, and an encoding alignment feature of each of the modalities included in the encoding alignment multi-modal feature is taken as a feature of the node corresponding to each of the modalities;

[0017] an edge is established between any two nodes, a feature of the edge is determined according to features of the two nodes, and the nodes and the features of the nodes and the edges and the features of the edges are taken as the graph data.

[0018] In an implementation manner, the feature of the edge is determined according to the features of the two nodes, and the determination includes:

[0019] a first cross-attention calculation result of one of the two nodes to the other node is determined, and a second cross-attention calculation result of the other node to one of the two nodes is determined;

[0020] the feature of the edge is obtained according to the first cross-attention calculation result and the second cross-attention calculation result.

[0021] In an implementation manner, according to the graph data, common features among the modalities and unique features of each of the modalities are obtained, and the obtaining includes:

[0022] features of the nodes and features of the edges included in the graph data are determined;

[0023] a self-attention calculation result of the node to the adjacent node of the node is determined according to the feature of the node and the feature of the adjacent node of the node, and the self-attention calculation result is recorded as a point self-attention calculation result;

[0024] a self-attention calculation result of the edge to the adjacent node of the edge is determined according to the feature of the node and the feature of the edge, and the self-attention calculation result is recorded as an edge self-attention calculation result;

[0025] the point self-attention calculation result and the edge self-attention calculation result are normalized respectively.

[0026] aggregating the self-attention calculation results of the normalized nodes to obtain neighborhood node feature information;

[0027] aggregating the self-attention calculation results of the normalized edges to obtain neighborhood edge feature information;

[0028] updating the features of the nodes according to the neighborhood node feature information and the neighborhood edge feature information to obtain common features between the multiple modalities and unique features of each modality.

[0029] In a second aspect, the embodiments of the present application also provide a multi-modal classification model, which comprises a multi-modal information encoding and alignment module, an adaptive multi-modal graph construction module, a decoupled graph learning module, and a decoupled feature fusion representation module connected in sequence.

[0030] The multi-modal information encoding and alignment module is configured to perform encoding and alignment processing on the original multi-modal medical data to obtain encoded and aligned multi-modal features.

[0031] The adaptive multi-modal graph construction module is configured to convert the encoded and aligned multi-modal features into graph data.

[0032] The decoupled graph learning module is configured to obtain common features between the multiple modalities and unique features of each modality according to the graph data.

[0033] The decoupled feature fusion representation module is configured to obtain a classification result of the original multi-modal medical data according to the common features and the unique features.

[0034] In an implementation manner, the total loss function of the multi-modal classification model is constructed in the following manner:

[0035] A loss function of the multi-modal information encoding and alignment module is determined and denoted as an encoding and alignment loss function.

[0036] A loss function of the decoupled graph learning module is determined and denoted as a decoupled loss function.

[0037] A loss function of the decoupled feature fusion representation module is determined and denoted as a classification loss function.

[0038] The total loss function is constructed according to the encoding and alignment loss function, the decoupled loss function, and the classification loss function.

[0039] In an implementation manner, the loss function of the decoupled graph learning module is determined and denoted as a decoupled loss function, which comprises:

[0040] acquiring common training features and specific training features of each modality output by the decoupling graph learning module;

[0041] constructing an intra-modality separation loss function according to the common training features and the specific training features of the same modality;

[0042] constructing an inter-modality separation loss function according to the specific training features of different modalities; and taking the intra-modality separation loss function and the inter-modality separation loss function as a decoupling loss function.

[0043] In a third aspect, an embodiment of the present application further provides a terminal device, wherein the terminal device comprises a memory, a processor, and a multi-modal medical data classification program stored in the memory and executable on the processor, and the processor implements the steps of the multi-modal medical data classification method when executing the multi-modal medical data classification program.

[0044] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a multi-modal medical data classification program, and the processor implements the steps of the multi-modal medical data classification method when executing the multi-modal medical data classification program.

[0045] Beneficial effects: The present application firstly performs encoding alignment processing on original multi-modal medical data to obtain multi-modal features after alignment, then converts the multi-modal features after alignment into graph data, then extracts common features between the multi-modal and specific features of each modality, and finally realizes classification of the original multi-modal medical data according to the common features and the specific features. Since the common features reflect the correlation between the multi-modal data, and the specific features can reflect the specificity of the modality, therefore, comprehensive utilization of the common features and the specific features can realize accurate classification of the multi-modal medical data. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a whole flowchart of the present application;

[0047] Figure 2 is a structure diagram of the multi-modal classification model in the embodiment of the present application;

[0048] Figure 3 is a schematic diagram of the multi-modal information encoding and alignment module in the embodiment of the present application;

[0049] Figure 4 is a structure diagram of the decoupling graph learning module in the embodiment of the present application;

[0050] Figure 5 is an internal structure principle block diagram of the terminal device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the present application will be described clearly and completely in combination with the embodiments and the accompanying drawings. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0052] It is found through research that diversified medical examinations can obtain multi-modal medical data of a patient, that is, one medical examination can obtain medical data of one modality. For example, medical data obtained through magnetic resonance imaging, computed tomography, ultrasonic examination and electrocardiogram examination constitutes multi-modal medical data. Through analysis and processing of the multi-modal medical data, classification of the multi-modal medical data can be realized, and the classification result can be used to reflect the disease degree of the patient. The prior art only classifies the multi-modal medical data, and does not fully consider the identity and specificity between the multi-modal medical data, thereby reducing the accuracy of the classification result of the multi-modal medical data.

[0053] To solve the above technical problems, the present application provides a multi-modal medical data classification method, model, device and storage medium, which solves the problem of reducing the accuracy of the classification result of the multi-modal medical data in the prior art. In specific implementation, first, original multi-modal medical data is obtained, and the original multi-modal medical data is encoded and aligned to obtain encoded and aligned multi-modal features. Then, the encoded and aligned multi-modal features are converted into graph data, the graph data including nodes and edges between any two nodes, wherein each node is used to represent each modality, the feature of the node corresponds to the feature of the modality, and the feature of the edge corresponds to the features of the two nodes connected by the edge. Then, according to the graph data, common features between the modalities and unique features of each modality are obtained. Finally, according to the common features and the unique features, a classification result of the original multi-modal medical data is obtained.

[0054] For example, magnetic resonance imaging data, computed tomography data, ultrasonic examination data and electrocardiogram examination data of a patient are collected, and the four modal data constitute original multi-modal medical data. The medical data of each modality is encoded and aligned, for example, the magnetic resonance imaging is encoded to extract the features of the magnetic resonance imaging. Since the features extracted from medical data of different modalities have different dimensions, the dimensions of the features of different modalities are adjusted to the same dimension to facilitate subsequent data processing.

[0055] The aligned multi-modal features are converted into graph data, each node of the graph data represents a modality, and an edge is established between any two nodes, for example, the four modalities correspond to four nodes, and an edge is established between any two nodes, thereby obtaining six edges, and the features of the edges reflect the features of the two nodes connected by the edges.

[0056] According to the graph data, common features and unique features of each modality can be obtained. Finally, according to the common features and the unique features, comprehensive classification of the four modalities, i.e., magnetic resonance imaging, computed tomography data, ultrasonic examination data, and electrocardiogram examination data, can be realized, which is used to represent whether the patient is ill and the degree of illness.

[0057] The multi-modal medical data classification method of the embodiment can be applied to a terminal device, which can be a terminal product with data processing function, such as a computer, etc. In the embodiment, as shown in Figure 1 The multi-modal medical data classification method specifically includes the following steps:

[0058] S100, obtaining original multi-modal medical data, and performing encoding alignment processing on the original multi-modal medical data to obtain encoded and aligned multi-modal features.

[0059] The original multi-modal medical data includes attribute data and medical examination data of a patient, wherein the attribute data includes gender, age, nationality, marital history, etc. The medical examination data includes MRI image examination results, PET image examination results, etc.

[0060] S200, converting the encoded and aligned multi-modal features into graph data, the graph data including nodes and edges between any two nodes, wherein each node is used to represent each modality, the features of the node correspond to the features of the modality, and the features of the edge correspond to the features of the two nodes connected by the edge;

[0061] S300, obtaining common features between the multi-modal and unique features of each modality according to the graph data;

[0062] S400, obtaining the classification result of the original multi-modal medical data according to the common features F sh and the unique features F sp .

[0063] In embodiment one, steps S100 to S400 are realized by a multi-modal classification model, as shown in Figure 2As shown, the multi-modal classification model includes a multi-modal information encoding and alignment module, an adaptive multi-modal graph construction module, a decoupled graph learning module, and a decoupled feature fusion representation module connected in sequence. The multi-modal information encoding and alignment module is used to implement step S100. The adaptive multi-modal graph construction module is used to implement step S200. The decoupled graph learning module is used to implement step S300. The decoupled feature fusion representation module is used to implement step S400, that is, the decoupled feature fusion representation module is used to splice the shared feature F sh and the unique feature F sp , and classify the spliced features, that is, classify the multi-modal medical data, and the classification result can be used as a reference for disease prediction.

[0064] The multi-modal classification model cannot be used to classify multi-modal medical data before training. After training, the multi-modal classification model can be used to classify multi-modal medical data. When training the multi-modal classification model, the parameters of the multi-modal classification model need to be adjusted by a loss function to obtain the trained multi-modal classification model. The total loss function (that is, the coordination loss function) for constructing the multi-modal classification model includes the following specific steps S01 to S06:

[0065] S01, determine the loss function of the multi-modal information encoding and alignment module, denoted as the encoding alignment loss function L a-cls :

[0066]

[0067] In the formula, M is the total number of modalities, is the aligned encoding feature of the mth modality output by the multi-modal information encoding and alignment module, y is the true classification label corresponding to the multi-modal medical data, and Loss (CrossEntropyLoss) is the cross-entropy loss. In this embodiment, the multi-modal information encoding and alignment module is guided by the true classification label for encoding and alignment, thereby improving the quality of encoding and alignment.

[0068] S02, obtain the shared training feature and the unique training feature of each modality output by the decoupled graph learning module.

[0069] S03, construct an intra-modality separation loss function L dif based on the shared training feature and the unique training feature of the same modality.

[0070]

[0071] and The common training features and the specific training features of the i-th modality, that is, the common training features and the specific training features of the same modality, and T represents a transposed matrix.

[0072] S04, constructing an inter-modality separation loss function L based on the specific training features of different modalities spa ; and taking the intra-modality separation loss function and the inter-modality separation loss function as a decoupling loss function.

[0073]

[0074] The specific training features of the i-th modality, The specific training features of the j-th modality, And The specific training features of the modality are constituted.

[0075] S05, determining the loss function of the decoupling feature fusion representation module, denoted as a classification loss function L b-cls :

[0076]

[0077] In the formula, y ij is a real classification label, p ij is the predicted classification probability output by the decoupling feature fusion representation module.

[0078] S06, constructing a total loss function L based on the encoding alignment loss function, the decoupling loss function, and the classification loss function:

[0079] L=α(L a-cls +L b-cls )+βL dif +γL spa

[0080] In the formula, α, β, and γ are all constants.

[0081] Embodiment two, based on embodiment one, the embodiment realizes step S100 through a multi-modality information encoding and alignment module, including the following specific steps: determining the type of each modality medical data in the original multi-modality medical data, and determining an encoder corresponding to each type, and extracting the features of the modality medical data through the encoder to obtain the encoding features of the modality medical data; adjusting the dimensions of the encoding features of the multi-modality medical data to the same dimension to obtain the encoding alignment multi-modality features.

[0082] The original multi-modal medical data includes numerical type original data and image type original data. For the numerical type original data, the numerical type encoder is used to encode the original data, and the numerical type original data can be encoded in binary. For the image type original data, the ResNet-101 with the same input size and image is used to extract the features of the image, and the features are used as the encoding of the image. After the encoding of the data of different modalities, the dimensions may be different, such as 1*128, 1*1024 and 1*512, and the features with different dimensions are input into the MLP, and the output dimension is designed to be 1024, so as to obtain 1*1024, 1*1024 and 1*1024, that is, the multi-modal information of the patient is extracted and aligned.

[0083] The structure of the multi-modal information encoding and alignment module is as shown in Figure 3 , that is, the original multi-modal medical data X m is first converted into encoded information , and then aligned into a new representation form

[0084]

[0085] In the formula, f m (·) represents the feature extraction branch for each modality data, and each modality data is encoded into a representation space. Align(·) represents a multi-modal alignment network, that is, the is mapped to a homogeneous representation space Z m is the encoded and aligned multi-modal feature, D a represents the dimension of the homogeneous space. And σ a (·) is designed to constrain the mapping process, which is a forward neural network constructed by three fully connected layers.

[0086] In the multi-modal alignment network Align(·), an attention mechanism is introduced to focus and fuse the multi-modal features through dynamic weights, realize the deep integration and semantic alignment between modalities, and the specific process is as follows:

[0087]

[0088] Above, σ(·) represents a fully connected network that maps the original multi-modal features to the same dimension, Q, K, and V represent Query, Key, and Value matrices, and W Q ,W K ,W V represent learnable parameters, d krepresents the dimension of the K matrix.

[0089] In embodiment three, based on embodiment one or embodiment two, the embodiment implements step S200 through the adaptive multi-modal graph construction module in embodiment one, including the following specific steps S201, S202 and S203:

[0090] S201, a node corresponding to each modality is set, and the encoding alignment feature of each modality contained in the encoding alignment multi-modal feature is set as the feature H of the node corresponding to each modality. v .

[0091] For example, four nodes corresponding to four modalities such as magnetic resonance imaging, computed tomography data, ultrasonic examination data and electrocardiogram examination data are set, and the feature of each modality is set as the feature of the corresponding node.

[0092] v is one of all edges V, represents a node set, represents an edge set.

[0093]

[0094] H v = σ v (Z)

[0095] S202, an edge is established between any two nodes, one of the two nodes v p is determined, the first cross-attention calculation result q of the other node v q to one of the nodes v p is determined, and the second cross-attention calculation result

[0096]

[0097] is the feature of the node v p , is the feature of the node v q , and CrossAtt(·) represents cross-attention calculation. S203, according to the first cross-attention calculation result and the second cross-attention calculation result, the feature H e of the edge is obtained, and the node and the feature of the node and the edge and the feature of the edge are taken as graph data.

[0098]

[0099]

[0100] In the formula, σ e ​​(·) represents edge feature mapping function, and symbol || represents concatenation.

[0101] Up to now, the graph data includes node features H v and edge features H e .

[0102] In the fourth embodiment, based on the first embodiment or the second embodiment or the third embodiment, the step S300 is implemented by the decoupled graph learning module in the first embodiment, and the specific steps S301 to S307 are as follows:

[0103] S301, determining the features of nodes and the features of edges contained in the graph data.

[0104] Let H v represent the features of all nodes, and let H e represent the features of all edges,

[0105] In the formula, G represents the graph data, and S represents the total number of edges.

[0106] S302, determining the self-attention calculation result of the node v i to the adjacent node v j of the node according to the features of the node v i and the features of the adjacent node v j of the node, denoted as point self-attention calculation result

[0107]

[0108] S303, determining the self-attention calculation result of the edge to the adjacent node of the edge according to the features of the node v i and the features of the edge e k , denoted as edge self-attention calculation result

[0109]

[0110] In the formula, are parameter matrices of the GNE network, and represent the point-to-point attention calculation network and the edge-to-point attention calculation network, respectively.

[0111] As shown in Figure 4 , the decoupled graph learning module at least includes a group of GNE networks, as shown in Figure 4 c graph, each group of GNE networks includes two GNE networks, and the structure of each GNE network is as shown in Figure 4As shown in the b diagram of FIG. 1, each GNE network includes a GNE module, a normalization layer and a nonlinear layer connected in sequence, and the structure of the GNE module is as shown in the a diagram of FIG. 1, that is, the GNE module includes an edge feature update layer and a point feature update layer connected in sequence. Figure 4 As shown in the b diagram of FIG. 1, each GNE network includes a GNE module, a normalization layer and a nonlinear layer connected in sequence, and the structure of the GNE module is as shown in the a diagram of FIG. 1, that is, the GNE module includes an edge feature update layer and a point feature update layer connected in sequence.

[0112] Each group of GNE networks updates the following formula: The updating manner is as follows:

[0113]

[0114] Where v p and v q represent two endpoints of the edge e k , and CrossAtt(a→b) represents the cross-attention calculation from a to b, where the definition of the cross-attention calculation is as follows:

[0115]

[0116]

[0117] The above H a ,H b represent two endpoints connected by an edge, and and the forms with subscripts represent Query, Key, Value matrices of the two endpoints, and Query, Key, Value are obtained from H*[W q ,W k ,W v ], where In the calculation of multi-head attention, d represents the dimension of the original feature, t represents the number of heads, represents a multi-head calculation matrix of different matrices, is a mapping after splicing of the multi-head, and the edge feature is obtained by the cross-attention calculation of the node feature and the conversion through the MLP.

[0118] S304, respectively, normalizing the point self-attention calculation result and the edge self-attention calculation result .

[0119] The point self-attention calculation result after normalization is denoted as α ij :

[0120]

[0121] a n represents a single-layer fully connected network, and σ(·) is an activation function, and the activation function includes ReLU.

[0122] The result of edge self-attention calculation after normalization denoted as η ik :

[0123]

[0124] S305, the point self-attention calculation result α after normalization. ij Aggregation processing is performed to obtain the feature information of neighboring nodes.

[0125]

[0126] S306, the normalized edge self-attention calculation result η ik Aggregation processing is performed to obtain neighborhood edge feature information.

[0127]

[0128] S307, based on the neighborhood node feature information and the neighborhood edge feature information Update the features of the nodes to obtain the common features F among the multimodalities. sh and the unique features F of each mode sp .

[0129] Each of the above GNE networks executes steps S302 to S306, that is, the next group of GNE networks will update the node features output by the previous group of GNE networks, take the output of one of the GNE networks in the last group of GNE networks as the final common feature, and take the output of the other GNE network in the last group of GNE networks as the final unique feature.

[0130] Steps S302 to S306 can be used to obtain the neighborhood node feature information output by the l-th group of GNE networks. and neighborhood edge feature information Then the neighbor node feature information output by the (l+1)th GNE network

[0131]

[0132] Neighboring node feature information It contains the common feature F output by the two GNE networks in the (l+1)th GNE network group. sh and unique features F sp .

[0133] In summary, the multi-modal information encoding and alignment module based on task guidance enables the application to make full use of the heterogeneity between multi-modal features; the point-edge graph learning enables the application to deeply mine and utilize the correlation between multi-modal information, and better perform disease analysis; and the decoupling learning strategy decouples the multi-modal features into modality-shared and modality-specific representations for disease diagnosis, so that the application can more effectively utilize the heterogeneity between multi-modal features and help improve the performance of the disease prediction model.

[0134] Based on the above embodiments, the application further provides a terminal device, a principle block diagram of which can be as shown in Figure 5 The terminal device includes a processor, a memory, a network interface and a display screen connected through a system bus. The processor of the terminal device is configured to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the terminal device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a multi-modal medical data classification method. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen.

[0135] Those skilled in the art can understand that Figure 5 The principle block diagram shown in the above embodiments is only a block diagram of part of the structure related to the application scheme, and does not constitute a limitation on the terminal device to which the application scheme is applied. Specifically, the terminal device can include more or fewer components than those shown in the diagram, or combine certain components, or have a different component arrangement.

[0136] In one embodiment, a terminal device is provided, which includes a memory, a processor and a multi-modal medical data classification program stored in the memory and executable on the processor. When the processor executes the multi-modal medical data classification program, the following operation instructions are implemented:

[0137] Obtaining original multi-modal medical data, and performing encoding and alignment processing on the original multi-modal medical data to obtain encoded and aligned multi-modal features;

[0138] Converting the encoded and aligned multi-modal features into graph data, the graph data including nodes and edges between any two nodes, wherein each node is configured to represent each modality, the feature of the node corresponds to the feature of the modality, and the feature of the edge corresponds to the features of the two nodes connected by the edge;

[0139] Obtaining common features between the multi-modal and specific features of each modality according to the graph data;

[0140] According to the common features and the unique features, a classification result of the original multi-modal medical data is obtained.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, storage, database or other medium used in the embodiments of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0142] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-modal medical data classification method, characterized by, The method comprises the following steps: obtaining original multi-modal medical data, and performing encoding alignment processing on the original multi-modal medical data to obtain encoded and aligned multi-modal features; converting the encoded and aligned multi-modal features into graph data, the graph data comprising nodes and edges between any two nodes, wherein each node is used to represent each modality, the feature of the node corresponds to the feature of the modality, and the feature of the edge corresponds to the features of the two nodes connected by the edge; obtaining common features between the modalities and specific features of each modality according to the graph data; obtaining a classification result of the original multi-modal medical data according to the common features and the specific features; the encoding alignment processing on the original multi-modal medical data to obtain the encoded and aligned multi-modal features comprises: determining the type of each modality medical data in the original multi-modal medical data, determining an encoder corresponding to each type, and extracting the feature of the modality medical data through the encoder to obtain the encoded feature of the modality medical data; adjusting the dimensions of the encoded features of the multi-modal medical data to the same dimension to obtain the encoded and aligned multi-modal features; the conversion of the encoded and aligned multi-modal features into graph data comprises: corresponding each modality to a node, and taking the encoded and aligned features of each modality contained in the encoded and aligned multi-modal features as the feature of the node corresponding to each modality; establishing an edge between any two nodes, and determining the feature of the edge according to the features of the two nodes, and taking the nodes, the features of the nodes and the edges and the features of the edges as the graph data.

2. The multi-modality medical data classification method of claim 1, wherein, the determination of the feature of the edge according to the features of the two nodes comprises: determining a first cross-attention calculation result of one of the two nodes to the other node, and determining a second cross-attention calculation result of the other node to one of the two nodes; obtaining the feature of the edge according to the first cross-attention calculation result and the second cross-attention calculation result.

3. The multi-modality medical data classification method of claim 1, wherein, the obtaining of the common features between the modalities and the specific features of each modality according to the graph data comprises: determining the features of the nodes and the features of the edges contained in the graph data; determining a self-attention calculation result of the node to the adjacent node of the node according to the feature of the node and the feature of the adjacent node of the node, denoted as a point self-attention calculation result; determining a self-attention calculation result of the edge to the adjacent node of the edge according to the feature of the node and the feature of the edge, denoted as an edge self-attention calculation result; respectively performing normalization processing on the point self-attention calculation result and the edge self-attention calculation result; performing aggregation processing on the point self-attention calculation result after normalization processing to obtain neighborhood node feature information; performing aggregation processing on the edge self-attention calculation result after normalization processing to obtain neighborhood edge feature information; updating the feature of the node according to the neighborhood node feature information and the neighborhood edge feature information to obtain the common features between the modalities and the specific features of each modality.

4. A multi-modal classification model, characterized in that, The multi-modal information encoding and alignment module, the adaptive multi-modal graph construction module, the decoupled graph learning module, and the decoupled feature fusion representation module are sequentially connected. The multi-modal information encoding and alignment module is configured to encode and align the original multi-modal medical data to obtain encoded and aligned multi-modal features, including: determining the type of each modality medical data in the original multi-modal medical data, determining an encoder corresponding to each type, extracting the feature of the modality medical data through the encoder to obtain the encoded feature of the modality medical data; and adjusting the dimension of the encoded features of the multi-modal medical data to the same dimension to obtain the encoded and aligned multi-modal features. The adaptive multi-modal graph construction module is configured to convert the encoded and aligned multi-modal features into graph data. The decoupled graph learning module is configured to obtain common features between the modalities and unique features of each modality according to the graph data. The decoupled feature fusion representation module is configured to obtain the classification result of the original multi-modal medical data according to the common features and the unique features. The conversion of the encoded and aligned multi-modal features into graph data includes: Each modality corresponds to a node, and the encoded and aligned features of each modality contained in the encoded and aligned multi-modal features are taken as the features of the node corresponding to each modality; An edge is established between any two nodes, and the features of the edge are determined according to the features of the two nodes, and the nodes and the features of the nodes and the edges and the features of the edges are taken as the graph data.

5. The multi-modal classification model of claim 4, wherein, The total loss function of the multi-modal classification model is constructed in the following manner: determining a loss function of the multi-modal information encoding and alignment module, denoted as an encoding alignment loss function : , M is the total number of modalities, is the aligned encoding feature of the mth modality output by the multi-modal information encoding and alignment module, and y is the real classification label corresponding to the multi-modal medical data; The loss function of the decoupled graph learning module is determined and denoted as a decoupled loss function; Determine the loss function of the decoupling feature fusion representation module, denoted as a classification loss function : , is a real classification label, is the predicted classification probability output by the decoupling feature fusion representation module; The total loss function is constructed according to the encoded and aligned loss function, the decoupled loss function, and the classification loss function; The loss function of the decoupled graph learning module is determined and denoted as a decoupled loss function, including: The common training features and the unique training features of each modality output by the decoupled graph learning module are obtained; The intra-modality separation loss function is constructed according to the common training features and the unique training features of the same modality; The inter-modality separation loss function is constructed according to the unique training features of different modalities; and the intra-modality separation loss function and the inter-modality separation loss function are taken as the decoupled loss function.

6. A terminal device, characterized by comprising: The terminal device includes a memory, a processor, and a multi-modal medical data classification program stored in the memory and executable on the processor. When the processor executes the multi-modal medical data classification program, the steps of the multi-modal medical data classification method according to any one of claims 1-3 are implemented.

7. A computer readable storage medium characterized in that, The computer-readable storage medium stores a multi-modal medical data classification program. When the multi-modal medical data classification program is executed by the processor, the steps of the multi-modal medical data classification method according to any one of claims 1-3 are implemented.

Images