Intelligent medical multimodal fusion method and system

Abstract

The present invention relates to the field of data processing technology, and in particular to an intelligent medical multimodal fusion method and system, the method comprising the following steps: S10, according to each patient's medical image and electronic medical record text, respectively obtain the image features, text features and diagnostic keyword codes corresponding to the patient, and fuse the image features and text features to obtain image text features; S20, take the image text features corresponding to a patient as the features of a node, and the diagnostic keyword codes as the features of the edges between nodes, so as to construct a graph convolutional network model; S30, for each node in the graph convolutional network model, fuse its own image text features with the image text features of neighboring nodes. The method of the present invention can utilize the correlation between multimodal information, so that the fused feature representation is more accurate, and then provide technical support for more accurate disease classification.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an intelligent medical multimodal fusion method and system. Background Art

[0002] With the development of artificial intelligence, its application in medicine is becoming more and more common. For example, neural networks are used to learn a large number of medical diagnostic images to achieve the purpose of disease classification. In order to improve the accuracy of disease classification, multimodal fusion technology has been further proposed, such as the fusion of image modality and text modality. Image and text fusion can supplement the missing information of a single image or text, making the classification ability of the neural network model stronger and the classification results more accurate.

[0003] However, most of the current multimodal fusion diagnosis solutions have one thing in common, that is, feature extraction is performed on the image modality and text modality, and then these features are trained separately, and finally classified through decision-level fusion. For example, the Chinese patent with patent number 2023116944924 discloses "a multimodal medical semantic alignment method for electronic medical records based on contrastive learning", which is to extract and train features of image and text modalities separately, and finally perform decision fusion. However, this method ignores the correlation between images and texts, that is, it separates the processing methods of image and text features. This separation leads to the loss of correlation between multimodal data, thereby reducing the accuracy of disease classification, or the current accuracy of disease classification can be further improved. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent medical multimodal fusion method and system, which can further improve the accuracy of disease classification results.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides an intelligent medical multimodal fusion method, comprising the following steps:

[0007] S10, according to the medical image and electronic medical record text of each patient, respectively obtain the image features, text features and diagnostic keyword codes corresponding to the patient, and fuse the image features and text features to obtain image text features;

[0008] S20, using the image text feature corresponding to a patient as a feature of a node, and the diagnosis keyword code as a feature of an edge between nodes, thereby constructing a graph convolutional network model;

[0009] S30, for each node in the graph convolutional network model, fusing its own image text features with the image text features of neighboring nodes.

[0010] In a more optimized implementation scheme, in S10, the medical images and electronic medical record text of each patient are processed according to the following steps:

[0011] S101, using a variational autoencoder to extract a hidden state in a medical image as the image feature;

[0012] S102, extracting a gender label and an age label from the electronic medical record text, and vectorizing the gender label and the age label using a text processing model to obtain gender features and age features as the text features;

[0013] S103, performing feature fusion on the hidden state, gender feature and age feature of the medical image to obtain the image text feature;

[0014] S104, extracting diagnostic keywords from the electronic medical record text, and encoding the diagnostic keywords by one-hot encoding to obtain the diagnostic keyword codes.

[0015] In the above scheme, age and gender are extracted from the electronic medical record text. On the one hand, age and gender characteristics are easy to obtain in electronic medical records and are stable, that is, each electronic medical record text contains age and performance; on the other hand, age and gender are relatively not private data; on the third hand, there is a certain correlation between age and gender and symptoms, which is a good indicator for the identification of diseases.

[0016] In a more optimized implementation, in S101, the obtained medical image is preprocessed before the hidden state is extracted from the medical image; and / or, in S102, the obtained electronic medical record text is preprocessed before the gender label and age label are extracted from the electronic medical record text. Preprocessing the medical image and the electronic medical record text can help extract more accurate information from them.

[0017] In a more optimized implementation scheme, in S30, the number of iterations is first set, and then in each iteration, for each node in the graph convolutional network model, the image text features of the node itself are fused with the image text features of the neighboring nodes. Through iterative training with a large amount of sample data, the graph convolutional network model after training can have a stronger analysis and recognition ability, and can classify diseases more accurately.

[0018] In a more optimized implementation scheme, the step of fusing the image text features of each node in the graph convolutional network model with the image text features of neighboring nodes includes:

[0019] S301, for each node in the graph convolutional network model, use the first formula to fuse its own image text features with the image text features of neighboring nodes to obtain a fused image text feature; the first formula is:

[0020]

[0021] S302, using a second formula to perform a second fusion of the image text feature after the first fusion with the image text feature of the self to obtain the image text feature after the second fusion; the second formula is:

[0022]

[0023] S303, updating the image text feature of the self with the image text feature after the secondary fusion;

[0024] Represents the image text features of node v at time layer t, represents the image text features of neighbor node w at time layer t, e vw represents the edge feature, |N(v)| represents the number of neighbor nodes of node v, and w vw represents the weight matrix between nodes v and w, LeakyReLu() is the activation function, represents the image text features of node v at time t+1, that is, the updated image text features of node v, and W represents all w vw A collection of weight matrices.

[0025] By integrating the features of neighboring nodes, nodes in the graph convolutional network model can better understand the position of the node in the graph and its relationship with other nodes, and learn key expressions from nodes with similar symptoms. By aggregating information from neighboring nodes, richer and more expressive node representations can be generated, making the node expression more representative. Obtaining neighboring node information helps improve the robustness of the model to local disturbances and noise.

[0026] Moreover, through multi-layer aggregation in S301, the information of distant nodes can be gradually transmitted to the target node, so that the node can perceive a wider range of graph information, and after the features of different neighbors are aggregated, the diversity and richness of the target node features can be increased; through S302, the aggregated features can be combined with the features of the node itself to obtain a richer node representation, and the multi-layer update enables the node features to integrate deeper and deeper information during the layer-by-layer transmission and adjustment.

[0027] In a second aspect, the present invention provides an intelligent medical multimodal fusion system, comprising:

[0028] The image text processing module is used to obtain the image features, text features and diagnostic keyword codes corresponding to each patient based on the medical image and electronic medical record text of each patient, and fuse the image features and text features to obtain the image text features;

[0029] A graph convolutional network model construction module, used to construct a graph convolutional network model by taking the image text feature corresponding to a patient as the feature of a node and the diagnosis keyword code as the feature of the edge between nodes;

[0030] The node feature fusion module is used to fuse the image and text features of each node in the graph convolutional network model with the image and text features of neighboring nodes.

[0031] In a third aspect, the present invention provides a computer program product, comprising computer-readable instructions, characterized in that the computer-readable instructions, when executed by a processor, implement the steps in the intelligent medical multimodal fusion method of the present invention.

[0032] In a fourth aspect, the present invention provides a computer-readable storage medium comprising computer-readable instructions, characterized in that the computer-readable instructions, when executed by a processor, implement the steps in the intelligent medical multimodal fusion method of the present invention.

[0033] In a fifth aspect, the present invention provides an electronic device, comprising: a memory storing program instructions; a processor connected to the memory, executing the program instructions in the memory, and implementing the steps in the intelligent medical multimodal fusion method of the present invention.

[0034] Compared with the prior art, the present invention has the following technical advantages:

[0035] The present invention relates to the fusion of two aspects. The first aspect of the fusion is to fuse the patient's medical image and the information in the electronic medical record text to form node features, which can enrich the expression of the patient's information. The second aspect of the fusion is to link the features of different nodes (patients) through diagnostic keywords, which can link the features of different patients. Therefore, the present invention can further improve the recognition and analysis capabilities of the trained graph convolutional network model and improve the accuracy of disease classification through the fusion of the two aspects.

[0036] For other advantages of the present invention, please refer to the relevant description in the embodiment section. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 This is an architecture diagram of the intelligent medical multimodal fusion system provided in an embodiment of the present invention.

[0039] Figure 2 The present invention provides a flowchart of a multi-modal fusion method for intelligent medical treatment.

[0040] Figure 3 for Figure 2 Detailed step-by-step diagram for S10.

[0041] Figure 4 for Figure 2 Detailed step diagram of S30.

[0042] Figure 5 The present invention provides a block diagram of the intelligent medical multimodal fusion system.

[0043] Figure 6 It is a block diagram of the composition of the electronic device provided in the embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] See also Figure 1 and Figure 2 , this embodiment provides an intelligent medical multimodal fusion method, comprising the following steps:

[0046] S10, according to each patient's medical image and electronic medical record text, respectively obtain the image features, text features and diagnostic keyword codes corresponding to the patient, and fuse the image features and text features to obtain image text features.

[0047] In this step, for each patient, the information of his or her own medical image and electronic medical record text is first fused to make the fused feature information more complete and avoid the loss of important information due to a single image or a single text.

[0048] S20, taking the image text feature corresponding to a patient as the feature of a node, and the diagnostic keyword code as the feature of the edge between nodes, so as to construct a graph convolutional network model.

[0049] In this step, one patient corresponds to one node, and the features of several patients are constructed into a graph convolutional network model, which makes it convenient to use the graph convolutional network to learn the features of each node in the graph convolutional network model and then mine the correlation between the nodes.

[0050] S30, for each node in the graph convolutional network model, fusing its own image text features with the image text features of neighboring nodes.

[0051] See also Figure 1 After obtaining the graph convolutional network model, a set number of iterative learning can be performed to improve the disease classification ability of the graph convolutional network. In each iteration, the same processing is performed on each node, that is, the image text features of the node itself are fused with the image text features of the neighboring nodes, and the image text features of the node itself are updated with the fused features.

[0052] See also Figure 3 As an example of a preferred implementation method, in S10, the medical images and electronic medical records of each patient are processed according to the following steps:

[0053] S101, using a variational autoencoder to extract hidden states in the medical image as the image features.

[0054] Medical images may be, for example, X-ray films, CT scan images, fundus photography, etc.

[0055] In order to improve the visual quality and readability of medical images, and thus improve the accuracy and reliability of medical diagnosis, the medical images can be preprocessed before extracting the hidden states in the medical images, including image enhancement and denoising.

[0056] When the variational autoencoder extracts the hidden state in the medical image, first, the preprocessed medical image x is input into the residual network (ResNet), and after being processed by multiple convolutional layers and pooling layers, the image feature f is obtained; then, the image feature f extracted by ResNet is input into the variational autoencoder, and the image feature f is mapped to the latent space through the encoder part, and the distribution parameters μ and σ of the latent space are generated, which represent the mean and variance of the latent space. Then, a random vector ε is obtained by sampling from the distribution of the latent space, and the random vector ε is combined with the distribution parameters μ and σ generated by the encoder using the reparameterization technique to calculate the hidden state z of the medical image. In the latent space, the hidden states corresponding to similar images will also be relatively close, which makes the hidden state continuous. The hidden state z is a more abstract and compact expression of the image features.

[0057] S102, extracting gender labels and age labels from the electronic medical record text, and vectorizing the gender labels and age labels using a text processing model to obtain gender features and age features as the text features.

[0058] In order to perform more accurate feature extraction, after obtaining the patient's electronic medical record text, the electronic medical record text can be preprocessed to eliminate noise, errors and non-standard content, making the text data more suitable for subsequent feature extraction.

[0059] The electronic medical record text contains a lot of information, but only gender characteristics and age characteristics are extracted in this step. There are many reasons for this. On the one hand, age and gender characteristics are easy to obtain in electronic medical records and are stable, that is, each electronic medical record text contains age and performance; on the other hand, age and gender are relatively not private data; on the other hand, there is a certain correlation between age and gender and symptoms, which is a good indicator for disease identification.

[0060] Extract patient-related gender and age labels from electronic medical records to facilitate vector conversion, and then merge with pathological image special diagnosis to form a richer and more complete patient feature vector. The text processing model can use the fastText model. First, decompose each sentence into a list of words or terms, and then use the fastText model to find the corresponding word vector for each word. Finally, perform a weighted average of the word vectors of all words in the sentence to obtain the vector representation of the entire sentence. Obtain the gender feature G and age feature A in the text.

[0061] S103, performing feature fusion on the hidden state, gender feature and age feature of the medical image to obtain the image text feature.

[0062] For the hidden state z of the medical image, combining the gender feature G and the age feature A for feature fusion can obtain more comprehensive and rich image and text fusion features, thereby achieving a more complete and vivid expression of the pathological scene.

[0063] The image text features obtained after fusion can be expressed as c u =[z, G, A].

[0064] S104, extracting diagnostic keywords from the electronic medical record text, and encoding the diagnostic keywords by one-hot encoding to obtain the diagnostic keyword codes.

[0065] The doctor's diagnostic keywords in the electronic medical record text are represented by one-hot encoding, where r is an integer greater than or equal to 1. For example, the electronic medical record text of eye films generally contains diagnostic keywords such as myopic atrophy arc, macular epiretinal membrane, retinal laser spot or arteriosclerosis. These common diagnostic keywords are encoded and represented, and each diagnostic keyword corresponds to a diagnostic keyword code.

[0066] See also Figure 1 In step S20, a patient is regarded as a node to construct a graph convolutional network model. The image text features corresponding to the patient are used as the features of the node, and the diagnostic keyword encoding is used as the feature of the edge between the nodes. The node can be represented by v, and the set of all nodes in the graph convolutional network model can be represented as a node set V. For each node, its feature is cu, that is, the image text features obtained after fusion of the patient corresponding to the node.

[0067] The edges between nodes in the graph convolutional network model are represented by (v i , r, v j ) indicates that v i 、v j They represent the connected nodes i and j respectively. The neighbor nodes of node v are represented by w, r is a vector of keyword encoding length, and the characteristics of the edge can be represented by e uw . Specifically, the feature of each edge can be expressed as a measure of the diagnostic similarity between patients. The features of the edges in the graph convolutional network model can help capture the associations and similarities between patients. For example, if node i has two diagnostic keywords, diabetes and hypertension, and node j has one diagnostic keyword, diabetes, then the encoding length is 2, and then the connection between these two nodes uses the common diabetes encoding as the edge feature. Neighboring nodes are connected by shared keywords, and the image hidden states z in the features of adjacent nodes are also similar, making it easier to find connections between similar diseases.

[0068] In step S30, in each iteration, each node will fuse its own image text features with the image text features of its neighboring nodes. Figure 4 , the specific processing process can be:

[0069] S301, for each node in the graph convolutional network model, use the first formula to fuse its own image text features with the image text features of neighboring nodes to obtain a fused image text feature; the first formula is:

[0070]

[0071] In the formula, Represents the image text features of node v at time layer t, represents the image text features of neighbor node w at time layer t, e vw represents the edge feature, |N(v)| represents the number of neighbor nodes of node v, and w vw Represents the weight matrix between nodes v and w. Each node can obtain additional information from neighboring nodes by fusing its own image text features with those of neighboring nodes at once.

[0072] Nodes actively transmit information to each other, mainly including: 1. Feature information: When a node transmits information to its neighboring nodes, it considers the feature information of the neighboring nodes and integrates this information into its own feature representation. 2. Relationship information: By considering the connection relationship between nodes, such as the weight of the edge (obtained through learning), the type of edge (the type of edge is determined by the diagnostic keyword, and the diagnostic keywords in the electronic medical record text are encoded through One-Hot as its edge type) and other information, we can better understand the interaction and influence relationship between nodes. 3. Local structural information: that is, the connection pattern between a node and its neighboring nodes (the connection pattern is mainly expressed by different edge types, that is, a node is connected to other nodes differently through different feature edges) helps the model better capture the local features and patterns of the graph convolutional network model.

[0073] The usual aggregation function discards the edge feature vector for aggregation due to the dimensionality problem of edge features. In this step, linking patient nodes through diagnostic keywords helps the model to aggregate and represent learning more accurately. The features of the edges can give each edge specific attributes (different diagnostic keywords), thereby enriching the expressiveness of node features. Edge features can contain contextual information between nodes, such as neighbor relationships between nodes, path features, etc. By considering the features of the edges, the model can better understand the position and role of nodes in the network, thereby performing aggregation and representation learning more accurately.

[0074] S302, using a second formula to perform a second fusion of the image text feature after the first fusion with the image text feature of the self to obtain the image text feature after the second fusion; the second formula is:

[0075]

[0076] Where LeakyReLu() is the activation function, represents the image text features of node v at time t+1, that is, the updated image text features of node v, and W represents all w vw The weight matrix is ​​a set of weight matrices composed of the parameters of the neural network layer. The weight matrix is ​​learned through the training process, and these weight matrices are updated through back propagation and gradient descent optimization methods to minimize the loss function.

[0077] The new feature representation obtained after the secondary fusion is obtained by applying the nonlinear activation function U t By applying activation functions, nodes can introduce nonlinear transformations to better capture the complex relationships and patterns of data and improve the expressiveness and generalization capabilities of graph convolutional network models. This feature update process enables nodes to dynamically adjust their feature representations based on their own features and the information of neighboring nodes.

[0078] S303: Update the image text features of the present invention with the image text features after the secondary fusion.

[0079] When the number of iterations reaches the set value, the obtained graph convolutional network model has a strong disease recognition ability and can accurately classify the disease. In actual application, the patient's medical image and electronic medical record text are processed according to the step S10, and then input into the graph convolutional network model, and the classification of the disease and the nodes of similar symptoms can be output for doctors to view.

[0080] See also Figure 5 Based on the same inventive concept, this embodiment also provides an intelligent medical multimodal fusion system, including an image and text processing module, a graph convolutional network model building module and a node feature fusion module.

[0081] The image text processing module is used to obtain the image features, text features and diagnostic keyword codes corresponding to each patient based on the medical image and electronic medical record text of each patient, and fuse the image features and text features to obtain the image text features;

[0082] The graph convolutional network model construction module is used to use the image text feature corresponding to a patient as the feature of a node, and the diagnosis keyword code as the feature of the edge between nodes, so as to construct a graph convolutional network model;

[0083] The node feature fusion module is used to fuse the image and text features of each node in the graph convolutional network model with the image and text features of neighboring nodes.

[0084] For more specific processing of each module, please refer to the relevant description of the aforementioned method, which will not be repeated here. For example, in the node feature fusion module, the specific processing of fusing the image text features of its own node with the image text features of neighboring nodes for each node in the graph convolutional network model is as follows: for each node in the graph convolutional network model, first use the first formula to fuse its own image text features with the image text features of neighboring nodes to obtain the image text features after the first fusion; then use the second formula to fuse the image text features after the first fusion with its own image text features for a second time to obtain the image text features after the second fusion; finally, update its own image text features with the image text features after the second fusion.

[0085] like Figure 6 As shown, this embodiment also provides an electronic device, which may include a processor 41 and a memory 42, wherein the memory 42 is coupled to the processor 41. It is worth noting that this figure is exemplary, and other types of structures may be used to supplement or replace this structure to achieve data extraction, report generation, communication or other functions.

[0086] like Figure 6 As shown, the electronic device may further include: an input unit 43, a display unit 44 and a power supply 45. It is worth noting that the electronic device does not necessarily have to include Figure 6 In addition, electronic devices may also include Figure 6 For components not shown, reference may be made to the prior art.

[0087] The processor 41 is sometimes also called a controller or an operation control, and may include a microprocessor or other processor devices and / or logic devices. The processor 41 receives inputs and controls the operations of various components of the electronic device.

[0088] The memory 42 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices, and may store information such as configuration information of the processor 41 and instructions executed by the processor 41. The processor 41 may execute the program stored in the memory 42 to implement information storage or processing. In one embodiment, the memory 42 also includes a buffer memory, i.e., a buffer, to store intermediate information.

[0089] The input unit 43 is used, for example, to input MRI images to the processor 41. The display unit 44 is used to display various results during the processing, and the display unit may be, for example, an LCD display, but the present invention is not limited thereto. The power supply 45 is used to provide power to the electronic device.

[0090] An embodiment of the present invention further provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed in an electronic device, the program product enables the electronic device to execute the operation steps included in the method of the present invention.

[0091] An embodiment of the present invention further provides a storage medium storing computer-readable instructions, wherein the computer-readable instructions enable an electronic device to execute the operation steps included in the method of the present invention.

[0092] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0093] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0094] Those of ordinary skill in the art will appreciate that the modules of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the constituent modules and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0096] The above-described embodiments are only specific implementations of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications, replacements and improvements within the technical scope disclosed by the present invention, and these modifications, replacements and improvements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An intelligent medical multimodal fusion method, characterized in that: The following steps are involved: S10, according to each patient's medical image and electronic medical record text, respectively obtain the image features, text features and diagnostic keyword codes corresponding to the patient, and fuse the image features and text features to obtain image text features; S20, using the image text feature corresponding to a patient as a feature of a node, and the diagnosis keyword code as a feature of an edge between nodes, thereby constructing a graph convolutional network model; S30, for each node in the graph convolutional network model, fusing its own image text features with the image text features of neighboring nodes; In S30, the number of iterations is first set, and then in each iteration, for each node in the graph convolutional network model, the image text features of the node itself are fused with the image text features of the neighboring nodes; The step of fusing the image text features of each node in the graph convolutional network model with the image text features of neighboring nodes includes: S301, for each node in the graph convolutional network model, use the first formula to fuse its own image text features with the image text features of neighboring nodes to obtain a fused image text feature; the first formula is: S302, using a second formula to perform a second fusion of the image text feature after the first fusion with the image text feature of the self to obtain the image text feature after the second fusion; the second formula is: S303, updating the image text feature of the self with the image text feature after the secondary fusion; Represents the image text features of node v at time layer t, represents the image text features of neighbor node w at time layer t, e υw represents the edge feature, |N(v)| represents the number of neighbor nodes of node v, and w vw represents the weight matrix between nodes v and w, LeakyReLu() is the activation function, represents the image text features of node v at time t+1, that is, the updated image text features of node v, and W represents all w vw A collection of weight matrices.

2. The intelligent medical multimodal fusion method according to claim 1, characterized in that: In S10, the medical images and electronic medical records of each patient are processed according to the following steps: S101, using a variational autoencoder to extract a hidden state in a medical image as the image feature; S102, extracting a gender label and an age label from the electronic medical record text, and vectorizing the gender label and the age label using a text processing model to obtain gender features and age features as the text features; S103, performing feature fusion on the hidden state, gender feature and age feature of the medical image to obtain the image text feature; S104, extracting diagnostic keywords from the electronic medical record text, and encoding the diagnostic keywords by one-hot encoding to obtain the diagnostic keyword codes.

3. The intelligent medical multimodal fusion method according to claim 2, characterized in that: In S101, before extracting the hidden state in the medical image, the obtained medical image is preprocessed; and / or, in S102, before extracting the gender label and age label in the electronic medical record text, the obtained electronic medical record text is preprocessed.

4. An intelligent medical multimodal fusion system, characterized in that: include: The image text processing module is used to obtain the image features, text features and diagnostic keyword codes corresponding to each patient based on the medical image and electronic medical record text of each patient, and fuse the image features and text features to obtain the image text features; A graph convolutional network model construction module, used to construct a graph convolutional network model by taking the image text feature corresponding to a patient as the feature of a node and the diagnosis keyword code as the feature of the edge between nodes; A node feature fusion module is used to fuse the image and text features of each node in the graph convolutional network model with the image and text features of neighboring nodes; In the node feature fusion module, for each node in the graph convolutional network model, the specific process of fusing its own image text features with the image text features of neighboring nodes is as follows: For each node in the graph convolutional network model, the first formula is first used to fuse the image text features of the node itself with the image text features of the neighboring node to obtain the image text features after the first fusion; then the second formula is used to fuse the image text features after the first fusion with the image text features of the node itself to obtain the image text features after the second fusion; finally, the image text features of the node itself are updated with the image text features after the second fusion; The first formula is: The second formula is: Represents the image text features of node v at time layer t, represents the image text features of neighbor node w at time layer t, e υw represents the edge feature, |N(v)| represents the number of neighbor nodes of node v, and w vw represents the weight matrix between nodes v and w, LeakyReLu() is the activation function, represents the image text features of node v at time t+1, that is, the updated image text features of node v, and W represents all w vw A collection of weight matrices.

5. A computer program product comprising computer readable instructions, characterized in that: When the computer-readable instructions are executed by a processor, the steps in the intelligent medical multimodal fusion method according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium comprising computer-readable instructions, characterized in that: When the computer-readable instructions are executed by a processor, the steps in the intelligent medical multimodal fusion method according to any one of claims 1 to 3 are implemented.

7. An electronic device, characterized in that: include: Memory, which stores program instructions; A processor is connected to the memory, executes program instructions in the memory, and implements the steps in the intelligent medical multimodal fusion method described in any one of claims 1-3.

Images