Multi-modal medical data classification method, model, equipment and storage medium

Abstract

The invention relates to the technical field of medical data processing, in particular to a multi-modal medical data classification method, model and equipment and a storage medium. The method comprises the following steps: firstly, carrying out coding alignment processing on original multi-modal medical data to obtain aligned multi-modal features, then converting the aligned multi-modal features into graph data, extracting common features among multiple modals and specific features of each modal, and finally, carrying out multi-modal feature extraction. Finally, the original multi-modal medical data are classified according to the common features and the specific features, the common features reflect the relevance between the multi-modal data, and the specific features can reflect the specificity of modals, so that accurate classification of the multi-modal medical data can be achieved by comprehensively utilizing the common features and the specific features.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data processing, and in particular to a multimodal medical data classification method, model, device and storage medium. Background Art

[0002] By performing various medical examinations on patients, multimodal medical data of the same patient can be obtained, that is, each medical examination can obtain a modality of medical data. For example, the medical data obtained from various examinations such as magnetic resonance imaging, computed tomography, ultrasound examination and electrocardiography constitute multimodal medical data. By analyzing and processing the multimodal medical data, the classification of the multimodal medical data can be achieved, and the classification results can be used to reflect the degree of illness of the patient. The existing technology simply classifies the multimodal medical data, and does not fully consider the similarity and specificity between the multimodal medical data, thereby reducing the accuracy of the classification results for the multimodal medical data.

[0003] In summary, existing technologies reduce the accuracy of classification results of multimodal medical data.

[0004] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a multimodal medical data classification method, model, device and storage medium, which solves the problem that the prior art reduces the accuracy of the classification results of multimodal medical data.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a multimodal medical data classification method, which comprises:

[0008] Acquiring original multimodal medical data, and performing coding alignment processing on the original multimodal medical data to obtain coding alignment multimodal features;

[0009] Converting the encoded aligned multimodal features into graph data, the graph data comprising nodes and edges between any two nodes, wherein each node is used to represent each modality, the feature of the node corresponds to the feature of the modality, and the feature of the edge corresponds to the features of the two nodes connected by the edge;

[0010] According to the graph data, the common features among multiple modalities and the unique features of each modality are obtained;

[0011] A classification result of the original multimodal medical data is obtained based on the common features and the unique features.

[0012] In one implementation, performing coding alignment processing on the original multimodal medical data to obtain coding aligned multimodal features includes:

[0013] Determine the type of each modality of medical data in the original multimodal medical data, determine an encoder corresponding to each of the types, and extract features of the modality of medical data through the encoder to obtain encoding features of the modality of medical data;

[0014] The dimensions of the encoded features of multimodal medical data are adjusted to the same dimension to obtain encoded aligned multimodal features.

[0015] In one implementation, converting the encoded aligned multimodal features into graph data includes:

[0016] Each modality corresponds to a node, and the encoding alignment feature of each modality included in the encoding alignment multimodal feature is used as the feature of the node corresponding to each modality;

[0017] An edge is established between any two nodes, and the characteristics of the edge are determined based on the characteristics of the two nodes, and the characteristics of the nodes and the characteristics of the edges are used as graph data.

[0018] In one implementation, determining the feature of the edge according to the features of the two nodes includes:

[0019] Determine a first cross-attention calculation result from one of the two nodes to the other node, and determine a second cross-attention calculation result from the other node to one of the nodes;

[0020] The feature of the edge is obtained according to the first cross-attention calculation result and the second cross-attention calculation result.

[0021] In one implementation, based on the graph data, common features among multiple modalities and unique features of each modality are obtained, including:

[0022] Determining the characteristics of the nodes and the characteristics of the edges contained in the graph data;

[0023] According to the characteristics of the node and the characteristics of the adjacent nodes of the node, a self-attention calculation result from the node to the adjacent nodes of the node is determined, which is recorded as the point self-attention calculation result;

[0024] Determine a self-attention calculation result from the edge to an adjacent node of the edge according to the feature of the node and the feature of the edge, recorded as an edge self-attention calculation result;

[0025] Normalizing the point self-attention calculation result and the edge self-attention calculation result respectively;

[0026] Aggregate the normalized point self-attention calculation results to obtain the neighborhood node feature information;

[0027] Aggregate the normalized edge self-attention calculation results to obtain the neighborhood edge feature information;

[0028] The features of the node are updated according to the neighborhood node feature information and the neighborhood edge feature information to obtain the common features between multiple modalities and the unique features of each modality.

[0029] In a second aspect, an embodiment of the present invention further provides a multimodal classification model, which includes a multimodal information encoding and alignment module, an adaptive multimodal graph construction module, a decoupled graph learning module, and a decoupled feature fusion representation module connected in sequence;

[0030] The multimodal information encoding and alignment module is used to perform encoding and alignment processing on the original multimodal medical data to obtain encoding and alignment multimodal features;

[0031] The adaptive multimodal graph construction module is used to convert the encoded aligned multimodal features into graph data;

[0032] The decoupled graph learning module is used to obtain the common features between multiple modalities and the unique features of each modality based on the graph data;

[0033] The decoupled feature fusion representation module is used to obtain the classification results of the original multimodal medical data based on the common features and the unique features.

[0034] In one implementation, the total loss function of the multimodal classification model is constructed by:

[0035] Determine the loss function of the multimodal information encoding and alignment module, denoted as the encoding alignment loss function;

[0036] Determine the loss function of the decoupling graph learning module, denoted as a decoupling loss function;

[0037] Determine the loss function of the decoupled feature fusion representation module, recorded as a classification loss function;

[0038] A total loss function is constructed based on the encoding alignment loss function, the decoupling loss function, and the classification loss function.

[0039] In one implementation, determining a loss function of the decoupled graph learning module, denoted as a decoupled loss function, includes:

[0040] Obtaining common training features and unique training features of each modality output by the decoupling graph learning module;

[0041] Construct the intra-modality separation loss function based on the common training features and unique training features of the same modality;

[0042] According to the unique training features of different modalities, an inter-modal separation loss function is constructed; and the intra-modal separation loss function and the inter-modal separation loss function are used as decoupling loss functions.

[0043] In a third aspect, an embodiment of the present invention further provides a terminal device, wherein the terminal device includes a memory, a processor, and a multimodal medical data classification program stored in the memory and executable on the processor, and when the processor executes the multimodal medical data classification program, the steps of the multimodal medical data classification method described above are implemented.

[0044] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a multimodal medical data classification program is stored. When the multimodal medical data classification program is executed by a processor, the steps of the multimodal medical data classification method described above are implemented.

[0045] Beneficial effects: The present invention first performs encoding alignment processing on the original multimodal medical data to obtain the aligned multimodal features, then converts the aligned multimodal features into graph data, and then extracts the common features between the multimodal data and the unique features of each modality. Finally, the original multimodal medical data is classified according to the common features and the unique features. Since the common features reflect the correlation between the multimodal data, and the unique features can reflect the specificity of the modality, the comprehensive use of the common features and the unique features can achieve accurate classification of the multimodal medical data. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is the overall flow chart of the present invention;

[0047] Figure 2 is a structural diagram of a multimodal classification model in an embodiment of the present invention;

[0048] Figure 3 is a schematic diagram of a multimodal information encoding and alignment module in an embodiment of the present invention;

[0049] Figure 4 is a structural diagram of a decoupling graph learning module in an embodiment of the present invention;

[0050] Figure 5 This is a block diagram of the internal structure principle of the terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following is a clear and complete description of the technical solution of the present invention in combination with the embodiments and the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Research has found that by conducting various medical examinations on patients, multimodal medical data of the same patient can be obtained, that is, each medical examination can obtain a modality of medical data. For example, the medical data obtained from various examinations such as magnetic resonance imaging, computed tomography, ultrasound examination and electrocardiography constitute multimodal medical data. By analyzing and processing multimodal medical data, classification of multimodal medical data can be achieved, and the classification results can be used to reflect the degree of illness of the patient. The existing technology simply classifies multimodal medical data, and does not fully consider the similarity and specificity between multimodal medical data, thereby reducing the accuracy of the classification results for multimodal medical data.

[0053] In order to solve the above technical problems, the present invention provides a multimodal medical data classification method, model, device and storage medium, which solves the problem that the prior art reduces the accuracy of the classification results of multimodal medical data. In specific implementation, the original multimodal medical data is first obtained, and the original multimodal medical data is coded and aligned to obtain coded aligned multimodal features; then the coded aligned multimodal features are converted into graph data, and the graph data includes nodes and edges between any two nodes, wherein each node is used to characterize each modality, the feature of the node corresponds to the feature of the modality, and the feature of the edge corresponds to the feature of the two nodes connected by the edge; then, based on the graph data, the common features between the multimodal and the unique features of each modality are obtained; finally, based on the common features and the unique features, the classification results of the original multimodal medical data are obtained.

[0054] For example, the patient's magnetic resonance imaging, computed tomography data, ultrasound data and electrocardiogram data are collected, and these four modal data constitute the original multimodal medical data. The medical data of each of the above modalities are encoded and aligned. For example, the magnetic resonance imaging is encoded to extract the features of the magnetic resonance imaging. Since the feature dimensions extracted from medical data of different modalities are different, in order to facilitate subsequent data processing, the feature dimensions of different modalities are adjusted to the same dimension.

[0055] The aligned multimodal features are then converted into graph data. Each node contained in the graph data represents a mode, and an edge is established between any two nodes. For example, the four modes mentioned above correspond to four nodes. When an edge is established between any two nodes, six edges are obtained. The characteristics of the edge reflect the characteristics of the two nodes connected by the edge.

[0056] Based on the graph data, we can get the features common to all four modalities (common features) and the unique features of each modality (unique features). Finally, based on the common features and unique features, we can achieve a comprehensive classification of the four modal data, namely, magnetic resonance imaging, computed tomography data, ultrasound data, and electrocardiogram data. This classification is used to characterize whether the patient is ill and the degree of illness.

[0057] The multimodal medical data classification method of this embodiment can be applied to a terminal device, which can be a terminal product with a data processing function, such as a computer. Figure 1 As shown in , the multimodal medical data classification method specifically includes the following steps:

[0058] S100, acquiring original multimodal medical data, and performing coding alignment processing on the original multimodal medical data to obtain coding aligned multimodal features.

[0059] The original multimodal medical data includes the patient's attribute data and medical examination data, where the attribute data includes gender, age, ethnicity, marital history, etc. The medical examination data includes MRI imaging results, PET imaging results, etc.

[0060] S200, converting the encoded aligned multimodal features into graph data, the graph data including nodes and edges between any two nodes, wherein each node is used to represent each modality, a feature of the node corresponds to a feature of the modality, and a feature of the edge corresponds to features of two nodes connected by the edge;

[0061] S300, obtaining common features among multiple modalities and unique features of each modality according to the graph data;

[0062] S400, based on the common feature F sh and the characteristic feature F sp , and obtain the classification result of the original multimodal medical data.

[0063] In the first embodiment, steps S100 to S400 are implemented by a multimodal classification model, such as Figure 2As shown, the multimodal classification model includes a multimodal information encoding and alignment module, an adaptive multimodal graph construction module, a decoupled graph learning module, and a decoupled feature fusion representation module connected in sequence; wherein the multimodal information encoding and alignment module is used to implement step S100; the adaptive multimodal graph construction module is used to implement step S200; the decoupled graph learning module is used to implement step S300; the decoupled feature fusion representation module is used to implement step S400, that is, the decoupled feature fusion representation module is used to splice the common feature F sh and unique feature F sp , and classify the concatenated features, that is, classify the multimodal medical data, and the classification results can be used as a reference for disease prediction.

[0064] The multimodal classification model cannot be used to classify multimodal medical data before training. The multimodal classification model can be used to classify multimodal medical data after training. When training a multimodal classification model, it is necessary to adjust the parameters of the multimodal classification model through a loss function to obtain a multimodal classification model after training. The total loss function (that is, the coordination loss function) of the multimodal classification model includes the following specific steps S01 to S06:

[0065] S01, determine the loss function of the multimodal information encoding and alignment module, denoted as the encoding alignment loss function L a-cls :

[0066]

[0067] Where M is the total number of modes, is the aligned encoding feature of the mth mode output by the multimodal information encoding and alignment module, y is the true classification label corresponding to the multimodal medical data, and Loss (CrossEntropyLoss) is the cross entropy loss. In this embodiment, the true classification label is used to guide the multimodal information encoding and alignment module to perform encoding alignment, thereby improving the quality of encoding alignment.

[0068] S02, obtaining the common training features and unique training features of each modality output by the decoupled graph learning module.

[0069] S03, based on the common training features of the same modality and unique training features Construct the intra-modal separation loss function L dif :

[0070]

[0071] and are the common training features and unique training features of the i-th modality, that is, the common training features and unique training features of the same modality, and T represents the transposed matrix.

[0072] S04, based on the unique training features of different modalities, construct the inter-modal separation loss function L spa ; and use the intra-modal separation loss function and the inter-modal separation loss function as decoupling loss functions.

[0073]

[0074] is the unique training feature of the i-th modality, is the unique training feature of the jth modality, and This constitutes the unique training characteristics of the modality.

[0075] S05, determining the loss function of the decoupled feature fusion representation module, denoted as the classification loss function L b-cls :

[0076]

[0077] In the formula, y ij is the true classification label, p ij It is the predicted classification probability output by the decoupled feature fusion representation module.

[0078] S06, constructing a total loss function L according to the encoding alignment loss function, the decoupling loss function, and the classification loss function:

[0079] L = α(L a-cls +L b-cls )+βL dif +γL spa

[0080] Where α, β, and γ are all constants.

[0081] Embodiment 2, based on embodiment 1, implements step S100 through a multimodal information encoding and alignment module, and includes the following specific steps: determining the type of each modality of medical data in the original multimodal medical data, and determining an encoder corresponding to each of the types, and extracting features of the modality medical data through the encoder to obtain encoding features of the modality medical data; adjusting the dimensions of the encoding features of the multimodal medical data to the same dimension to obtain encoding aligned multimodal features.

[0082] The original multimodal medical data includes raw data of numerical type and raw data of image type. For raw data of numerical type, the raw data is encoded using a numerical encoder, for example, the raw data of numerical type can be encoded using binary. For raw data of image type, ResNet-101 with the same input size as the image is used to extract the features of the image, and the features are used as the encoding of the image. After encoding, data of different modalities may have different dimensions, for example, there may be features of different dimensions such as 1*128, 1*1024, and 1*512. The features of different dimensions are input into the MLP, and its output dimension is designed to be a common 1024, thereby obtaining three high-level features of the same dimensions of 1*1024, 1*1024, and 1*1024, that is, the multimodal information of the patient is extracted after encoding and alignment.

[0083] The structure of the multimodal information encoding and alignment module is as follows: Figure 3 As shown, that is, the original multimodal medical data X m First converted into coded information Then aligned into a new representation

[0084]

[0085] In the formula, f m (·) represents the feature extraction branch for each modality data, encoding the data of each modality into a representation space. Align(·) represents the multimodal alignment network, that is, Mapping to a homogeneous representation space Z m To encode the aligned multimodal features, D a represents the dimension of the homogeneous space. And σ a (·) is designed to constrain this mapping process. It is a feed-forward neural network constructed by three fully connected layers.

[0086] In the multimodal alignment network Align(·), an attention mechanism is introduced to focus and fuse multimodal features through dynamic weights to achieve deep integration and semantic alignment between modalities. The specific process is as follows:

[0087]

[0088] Above, σ(·) represents the original multimodal features Mapped to a fully connected network of the same dimension, Q, K, V represent the Query, Key, and Value matrices respectively, and W Q ,W K ,W V represents the learnable parameters, d kRepresents the dimension of the K matrix.

[0089] Embodiment 3, based on embodiment 1 or embodiment 2, this embodiment implements step S200 through the adaptive multimodal graph construction module in embodiment 1, including the following specific steps S201, S202, and S203:

[0090] S201, each modality corresponds to a node, and the encoding alignment feature of each modality included in the encoding alignment multimodal feature is used as the feature H of the node corresponding to each modality v .

[0091] For example, the four modalities of magnetic resonance imaging, computed tomography data, ultrasound examination data, and electrocardiogram examination data correspond to four nodes, and the features of each modality are used as the features of the corresponding nodes.

[0092] v is one of the edges in V, Represented as a node set, Represented as an edge set.

[0093]

[0094] H v =σ v (Z)

[0095] S202, establish an edge between any two nodes, determine one of the two nodes v p To another node v q The first cross-attention calculation results And determine another node v q To one of the nodes v p The second cross-attention calculation result

[0096]

[0097] For node v p Features, For node v q The feature of , CrossAtt(·) represents the cross-attention calculation.

[0098] S203, obtaining the feature H of the edge according to the first cross-attention calculation result and the second cross-attention calculation result e , and use the nodes and the features of the nodes as well as the edges and the features of the edges as graph data.

[0099]

[0100] In the formula, σ e(·) represents the edge feature mapping function, and the symbol || represents concatenation.

[0101] So far, the graph data includes the node features H v and edge feature H e .

[0102] Embodiment 4, based on embodiment 1 or embodiment 2 or embodiment 3, this embodiment implements step S300 through the decoupling graph learning module in embodiment 1, including the following specific steps S301 to S307:

[0103] S301, determining the characteristics of the nodes and the characteristics of the edges contained in the graph data.

[0104] The node feature matrix H v Represent the characteristics of all nodes, using the edge feature matrix H e represents the characteristics of all edges,

[0105] In the formula, G represents the graph data and S represents the total number of edges.

[0106] S302, based on the node v i The characteristics of the node and the adjacent nodes v j The characteristics of the node v i The adjacent node v to this node j The self-attention calculation result of point is recorded as the self-attention calculation result of point

[0107]

[0108] S303, based on the node v i The characteristics and edges k The feature of the edge is used to determine the self-attention calculation result of the adjacent node of the edge, which is recorded as the edge self-attention calculation result

[0109]

[0110] In the formula, are the parameter matrices of the GNE network, and They represent point-to-point attention calculation and edge-to-point attention calculation networks respectively.

[0111] like Figure 4 As shown in , the decoupled graph learning module includes at least a set of GNE networks, such as Figure 4 As shown in Figure c, each group of GNE networks includes two GNE networks, and the structure of each GNE network is as follows Figure 4As shown in Figure b, each GNE network includes a GNE module, a normalization layer, and a nonlinear layer connected in sequence, and the structure of the GNE module is as follows Figure 4 As shown in Figure a, the GNE module includes an edge feature update layer and a point feature update layer connected in sequence.

[0112] Each group of GNE networks is To update, the update method is as follows:

[0113]

[0114] where v p and v q Respectively represent the edge e k The two endpoints of , CrossAtt(a→b) represents the cross-attention calculation from a to b, where the cross-attention calculation is defined as follows:

[0115]

[0116]

[0117] Above H a ,H b represents the two endpoints connected by the edge, and The subscripted form represents the query, key, and value matrices of the two endpoints. Query, Key, and Value are represented by H*[W q ,W k ,W v ] to obtain, among which In the calculation of multi-head attention, d represents the dimension of the original feature, t represents the number of heads, A multi-head calculation matrix representing different matrices, For the mapping of multiple heads after splicing, the edge features are calculated from the node features through cross-attention and transformed by MLP.

[0118] S304, respectively calculate the self-attention results of the points And the edge self-attention calculation results Perform normalization.

[0119] Point self-attention calculation results after normalization Denoted as α ij :

[0120]

[0121] a n represents a single-layer fully connected network, σ(·) is the activation function, and the activation function includes ReLU.

[0122] The calculation results of edge self-attention after normalization Denoted as η ik :

[0123]

[0124] S305, calculate the point self-attention result α after normalization ij Perform aggregation processing to obtain neighborhood node feature information

[0125]

[0126] S306, calculate the edge self-attention result η after normalization ik Perform aggregation processing to obtain neighborhood edge feature information

[0127]

[0128] S307, based on the neighborhood node feature information and the neighborhood edge feature information Update the features of the nodes to obtain the common features F between multiple modalities sh and the characteristic features F of each mode sp .

[0129] Each of the above-mentioned groups of GNE networks executes steps S302 to S306, that is, the next group of GNE networks will update the node features output by the previous group of GNE networks, and use the results output by one of the GNE networks in the last group of GNE networks as the final common features, and use the results output by another GNE network in the last group of GNE networks as the final unique features.

[0130] By adopting steps S302 to S306, the neighborhood node feature information output by the first group of GNE networks can be obtained. and neighborhood edge feature information Then the neighborhood node feature information output by the l+1th group of GNE networks is

[0131]

[0132] Neighborhood node feature information Contains the common features F output by the two GNE networks in the l+1th group of GNE networks sh and unique feature F sp .

[0133] In summary, the task-guided multimodal information encoding and alignment module of the present invention enables the present invention to more fully utilize the heterogeneity between multimodal features; the point-edge graph learning enables the present invention to deeply explore and utilize the correlation between multimodal information and better perform disease analysis; the decoupling learning strategy decouples multimodal features into modality sharing and modality-specific representations for disease diagnosis, enabling the present invention to more effectively utilize the heterogeneity between multimodal features and help improve the performance of disease prediction models.

[0134] Based on the above embodiments, the present invention further provides a terminal device, whose principle block diagram can be shown as follows: Figure 5 As shown. The terminal device includes a processor, a memory, a network interface, and a display screen connected via a system bus. Among them, the processor of the terminal device is used to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a multimodal medical data classification method is implemented. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen.

[0135] Those skilled in the art will understand that Figure 5 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal device to which the scheme of the present invention is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0136] In one embodiment, a terminal device is provided. The terminal device includes a memory, a processor, and a multimodal medical data classification program stored in the memory and executable on the processor. When the processor executes the multimodal medical data classification program, the following operation instructions are implemented:

[0137] Acquiring original multimodal medical data, and performing coding alignment processing on the original multimodal medical data to obtain coding alignment multimodal features;

[0138] Converting the encoded aligned multimodal features into graph data, the graph data comprising nodes and edges between any two nodes, wherein each node is used to represent each modality, the feature of the node corresponds to the feature of the modality, and the feature of the edge corresponds to the features of the two nodes connected by the edge;

[0139] According to the graph data, the common features among multiple modalities and the unique features of each modality are obtained;

[0140] A classification result of the original multimodal medical data is obtained based on the common features and the unique features.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal medical data classification method, characterized in that: include: Acquiring original multimodal medical data, and performing coding alignment processing on the original multimodal medical data to obtain coding alignment multimodal features; Converting the encoded aligned multimodal features into graph data, the graph data comprising nodes and edges between any two nodes, wherein each node is used to represent each modality, the feature of the node corresponds to the feature of the modality, and the feature of the edge corresponds to the features of the two nodes connected by the edge; According to the graph data, the common features among multiple modalities and the unique features of each modality are obtained; A classification result of the original multimodal medical data is obtained based on the common features and the unique features.

2. The multimodal medical data classification method according to claim 1, characterized in that: The encoding alignment processing is performed on the original multimodal medical data to obtain the encoding alignment multimodal features, including: Determine the type of each modality of medical data in the original multimodal medical data, determine an encoder corresponding to each of the types, and extract features of the modality of medical data through the encoder to obtain encoding features of the modality of medical data; The dimensions of the encoded features of multimodal medical data are adjusted to the same dimension to obtain encoded aligned multimodal features.

3. The multimodal medical data classification method according to claim 1, characterized in that: Converting the encoded aligned multimodal features into graph data includes: Each modality corresponds to a node, and the encoding alignment feature of each modality included in the encoding alignment multimodal feature is used as the feature of the node corresponding to each modality; An edge is established between any two nodes, and the characteristics of the edge are determined based on the characteristics of the two nodes, and the characteristics of the nodes and the characteristics of the edges are used as graph data.

4. The multimodal medical data classification method according to claim 3, characterized in that: The step of determining the feature of the edge according to the features of the two nodes includes: Determine a first cross-attention calculation result from one of the two nodes to the other node, and determine a second cross-attention calculation result from the other node to one of the nodes; The feature of the edge is obtained according to the first cross-attention calculation result and the second cross-attention calculation result.

5. The multimodal medical data classification method according to claim 1, characterized in that: Based on the graph data, the common features among multiple modalities and the unique features of each modality are obtained, including: Determining the characteristics of the nodes and the characteristics of the edges contained in the graph data; According to the characteristics of the node and the characteristics of the adjacent nodes of the node, a self-attention calculation result from the node to the adjacent nodes of the node is determined, which is recorded as the point self-attention calculation result; Determine a self-attention calculation result from the edge to an adjacent node of the edge according to the feature of the node and the feature of the edge, recorded as an edge self-attention calculation result; Normalizing the point self-attention calculation result and the edge self-attention calculation result respectively; Aggregate the normalized point self-attention calculation results to obtain the neighborhood node feature information; Aggregate the normalized edge self-attention calculation results to obtain the neighborhood edge feature information; The features of the node are updated according to the neighborhood node feature information and the neighborhood edge feature information to obtain the common features between multiple modalities and the unique features of each modality.

6. A multimodal classification model, characterized in that: It includes a multimodal information encoding and alignment module, an adaptive multimodal graph construction module, a decoupled graph learning module, and a decoupled feature fusion representation module that are connected in sequence; The multimodal information encoding and alignment module is used to perform encoding and alignment processing on the original multimodal medical data to obtain encoding and alignment multimodal features; The adaptive multimodal graph construction module is used to convert the encoded aligned multimodal features into graph data; The decoupled graph learning module is used to obtain the common features between multiple modalities and the unique features of each modality based on the graph data; The decoupled feature fusion representation module is used to obtain the classification results of the original multimodal medical data based on the common features and the unique features.

7. The multimodal classification model according to claim 6, characterized in that: The total loss function of the multimodal classification model is constructed in a manner including: Determine the loss function of the multimodal information encoding and alignment module, denoted as the encoding alignment loss function; Determine the loss function of the decoupling graph learning module, denoted as a decoupling loss function; Determine the loss function of the decoupled feature fusion representation module, recorded as a classification loss function; A total loss function is constructed based on the encoding alignment loss function, the decoupling loss function, and the classification loss function.

8. The multimodal classification model according to claim 7, wherein: Determine the loss function of the decoupled graph learning module, denoted as the decoupled loss function, including: Obtaining common training features and unique training features of each modality output by the decoupling graph learning module; Construct the intra-modality separation loss function based on the common training features and unique training features of the same modality; According to the unique training features of different modalities, an inter-modal separation loss function is constructed; and the intra-modal separation loss function and the inter-modal separation loss function are used as decoupling loss functions.

9. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a multimodal medical data classification program stored in the memory and executable on the processor. When the processor executes the multimodal medical data classification program, the steps of the multimodal medical data classification method as described in any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multimodal medical data classification program, and when the multimodal medical data classification program is executed by a processor, the steps of the multimodal medical data classification method according to any one of claims 1 to 5 are implemented.

Images