A medical intelligent dialogue method based on deep learning

Abstract

The present invention discloses a medical intelligent dialogue method based on deep learning, which relates to the field of intelligent medical technology. The present invention constructs a rare disease knowledge graph. Even when the training data is insufficient, the implicit connection between the disease and the symptoms can be deduced through the correlation relationship in the graph, which greatly enhances the ability to handle complex cases such as rare diseases. Through the graph reasoning mechanism, a more accurate list of rare diseases can be provided, and relevant diagnostic solutions can be reasonably and preferentially displayed to users to reduce the risk of misjudgment. Through the multimodal Transformer model, multi-source data such as medical records, medical images and laboratory test results can be effectively integrated to generate a unified comprehensive feature representation. The deep integration has the ability to comprehensively analyze the patient's health status from different angles, thereby improving the accuracy of diagnosis. Based on the preliminary diagnosis results, the order of inquiries and the depth of questions can be adjusted according to the specific conditions of different patients, providing a more accurate and personalized inquiry experience.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical technology, and in particular to a medical intelligent dialogue method based on deep learning. Background Art

[0002] Rare diseases refer to diseases with extremely low incidence rates. There are more than 7,000 known rare diseases worldwide, but the number of individual cases is small, and the causes, symptoms and treatment options of each disease are different. Data on these diseases are scarce and the conditions are highly complex.

[0003] Current medical intelligent dialogue methods rely on large-scale labeled data for training, but the number of cases of rare diseases is extremely small, and there is a lack of sufficient training data. The model finds it difficult to capture the characteristics of rare diseases, resulting in insufficient generalization capabilities in the diagnosis of rare diseases. In addition, rare diseases are usually accompanied by multiple causes, such as genetic factors, environmental influences, etc. The diagnostic data of rare diseases is also difficult to effectively use data to prescribe the right medicine during inference and processing due to the complex causes and diverse manifestations.

[0004] Traditional medical intelligent dialogue solutions, in response to the scarcity of rare disease data, mostly rely on existing medical knowledge to reason and diagnose rare diseases. They can provide a certain degree of accuracy in some scenarios, but their scalability is limited. They may expand the data set to cover more rare disease scenarios through data augmentation, but this cannot make up for the inherent shortage of rare disease samples. The generated data is also likely to differ significantly from actual cases, resulting in low reasoning and diagnostic accuracy. Or they may use expert systems. Although expert systems are effective in some rare disease scenarios, their rule bases are slow to update, making it difficult to cope with the rapidly changing medical field. Therefore, there is an urgent need for a medical intelligent dialogue method based on deep learning to solve such problems. Summary of the Invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides a medical intelligent dialogue method based on deep learning to solve the problems faced by traditional medical intelligent dialogue solutions when dealing with rare diseases and utilizing knowledge graphs, such as data scarcity, insufficient knowledge coverage, and limited reasoning capabilities.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] The present invention provides a medical intelligent dialogue method based on deep learning, which includes:

[0009] Step S1: construct a rare disease knowledge graph.

[0010] Based on existing literature, clinical guidelines, expert knowledge, and data from medical research, we structure information on the causes, symptoms, diagnostic methods, treatment options, and drug interactions of rare diseases to generate a knowledge graph. In the absence of sufficient data training, the knowledge graph serves as a knowledge base for preliminary diagnosis, providing a list of potential rare diseases associated with symptoms.

[0011] Step S2, integrated multimodal data processing,

[0012] Multimodal data, including medical records, medical images, and laboratory test results, is introduced and processed using a multimodal Transformer model. Node information in the knowledge graph is used as prior knowledge and jointly encoded and analyzed with the input multimodal data to generate a comprehensive understanding of the multimodal information, including feature extraction results based on patient symptoms, images, and genetic data, namely multimodal features.

[0013] Step S3, knowledge graph reasoning and symptom matching,

[0014] Use knowledge graphs for symptom matching and combine multimodal data for preliminary diagnosis;

[0015] Step S4: Dialogue management and patient personalization.

[0016] The order and content of the inquiries are dynamically adjusted based on the diagnosis results obtained in step S3, and the depth and direction of the questions are adjusted for patients with different conditions.

[0017] Furthermore, the knowledge graph contains nodes for symptoms, diseases, drugs, and diagnostic methods, and the knowledge graph serves as the core knowledge source of the dialogue system.

[0018] Furthermore, in step S1, the method of constructing a rare disease knowledge graph is as follows:

[0019] Based on literature, clinical guidelines and expert knowledge, collect relevant data on rare diseases, including causes, symptoms, diagnostic methods, treatment options and drug interactions, and define the data matrix D∈R m×n , where: D = {d ij}, d ij Represents the jth category of information for the i-th rare disease, including causes, symptoms, and drugs. m is the total number of rare diseases, and n is the number of categories of information related to each disease. The heterogeneous data is normalized using the normalization function:

[0020] in, is the normalized data, μ j is the mean of the j-th type of information, σ j is the standard deviation of the j-th category information, for each rare disease Define the following nodes:

[0021] S i ={s1,s2,…,s p} indicates disease Related p symptoms,

[0022] Expression and disease Related q treatment methods,

[0023] M i ={m1,m2,…,m r} indicates disease Related r drugs,

[0024] The relationship between nodes is represented by the relationship matrix R∈R k×k express: Among them, R ij Indicates disease and diseases The strength of the relationship between , f(·) represents the relationship definition function based on similarity, using the Jaccard similarity coefficient;

[0025] Use graph embedding method to embed nodes and relationships in knowledge graph into vector space, let for each node v i , and its corresponding vector is represented as v i ∈R d , d is the embedding dimension, then the relationship r in the knowledge graph ij The embedding representation is: ij =v i +rv j , where r ij Indicates disease and diseases The relationship between them is represented by the embedding of r, v i and v j Diseases and The node vector representation of the embedded node v i And the relationship r, deduce the possibility of potential disease, and define the inference probability function: in, When the embedding representation S of a given symptom set is expressed, the disease The diagnostic probability of S is the embedded representation of the given symptom set. Based on the inference result, the most likely diagnosis result of the rare disease is output;

[0026] Diagnosis results The score is: in, Indicates disease With input symptoms kThe matching degree, ω k Symptoms k The weight of the disease is used to sort and output a potential list of rare diseases through the matching degree of comprehensive symptoms and diseases.

[0027] Furthermore, in step S2, a multimodal Transformer model is used to jointly encode and analyze different data types such as medical records, medical images, and laboratory test results:

[0028] Process medical record text data, extract symptoms, diagnosis and medical history information, and set the medical record text sequence as T 1 ={t1,t2,…,t n}, where t i The word vector embedding representing the i-th word is converted into a high-level semantic representation through the self-attention mechanism: in, Represents the high-dimensional semantic representation of medical record text, d is the embedding dimension, are the query, key, and value matrices of the text sequence, respectively, d k is the dimension of each attention head, T represents the transposition operation, and n is the length of the medical record text;

[0029] For medical image data, convolutional neural network CNN is used for feature extraction. The input medical image is a three-dimensional matrix I∈R h×w×c , where h and w are the height and width of the image, and c is the number of channels. Through the CNN network, it is converted into a low-dimensional feature representation: F I =CNN(I), where F I ∈R m×d represents the extracted image features, m is the number of spatial locations in the feature map, and d is the embedding dimension;

[0030] Laboratory test results are represented by a vector L = {l1,l2,…,l k}, where l i The result of the i-th detection is mapped to the feature space through the fully connected layer: F L =W L L+b L , where F L ∈R d Represents the characteristic representation of laboratory test results, W L ∈R d×k is the weight matrix, b L ∈R d is the bias term, k is the number of detection items, and d is the embedding dimension.

[0031] Furthermore, in step S2, a multimodal Transformer model is used, and the features of the three types of data are HT 、F I and F L , fused through a multimodal self-attention mechanism:

[0032] Where Z∈R (n+m+1)×d is the fused multimodal feature representation, where n, m, and 1 correspond to the number of features of text, image, and laboratory test data, respectively, and Q M ,K M ,V M ∈R (n+m+1)×d are the fused query, key, and value matrices respectively, and d is the embedding dimension, where:

[0033] Q M =[H T ; F I ; F L ]W Q , K M =[H T ; F I ; F L ]W K , V M =[H T ; F I ; F L ]W V , where [H T ; F I ; F L ] represents the concatenation of text, image and laboratory test features, W Q 、W K 、W V ∈R d×d The mapping matrices for query, key, and value respectively;

[0034] The fused multimodal feature Z is obtained through the multimodal self-attention mechanism, and global pooling is performed to generate the final comprehensive feature vector F M :F M =GlobalPool(Z), where F M ∈R d is a multimodal feature vector used for subsequent diagnostic reasoning and symptom matching. The multimodal feature vector F M As the result of comprehensive understanding of multimodal information; realize feature extraction and comprehensive understanding of multimodal data.

[0035] Furthermore, in step S3, based on the knowledge graph generated in step S1, the symptoms input by the patient and the multimodal features are matched through the knowledge reasoning mechanism to output a preliminary diagnosis result, including a list of possible diseases, cause speculation and preliminary treatment recommendations.

[0036] Furthermore, in step S3, the patient's input symptoms and multimodal features are matched with the knowledge graph generated in step S1 through the knowledge reasoning mechanism, and a preliminary diagnosis result is output:

[0037] Suppose rare disease knowledge graph G = (V, E), where V is a node set representing disease, symptom, drug, and diagnostic method entities, and E is an edge set representing the relationship between entities. For each node v i ∈V, using vector v i ∈R d Represents, where d is the embedding dimension, and the node embedding formula is: i =f(v i ), where v i is the node v in the knowledge graph i The embedding vector, f(v i ) is to put the node v i Functions that convert to embedding vectors, which are converted by graph embedding methods such as TransE or GraphSAGE;

[0038] Multimodal feature F M A comprehensive vector representing the patient's input symptoms, medical imaging, and laboratory test results, with the same dimension as the knowledge graph node embedding v i Same, multimodal feature F M Match with nodes in the knowledge graph, F M =GlobalPool(Z), where F M is the fused multimodal feature vector, representing the comprehensive information of the patient, and Z is the feature matrix after fusing text, image, and laboratory data.

[0039] Furthermore, in step S3, symptoms and diseases are matched.

[0040] In the knowledge graph, there is an edge between disease and symptoms. ij ∈E, the multimodal features F of the patient input M and the disease node v in the graph i Perform matching and calculate the similarity between symptoms and diseases. The matching formula is:

[0041] Among them, S(F M ,v i ) represents the patient’s multimodal features F M With disease node v i The cosine similarity of |F is used to evaluate the matching degree between the input symptoms and the disease. M | and |v i |F M and v i The vector norm of , is the inner product of two vectors;

[0042] Define a relationship-based reasoning model, deduce the relationships in the knowledge graph, and calculate the matching degree of potential diseases. The reasoning formula is: in, Represents the patient's multimodal features F M Matching diseases The probability of S(F M ,v j ) is the multimodal feature and disease The similarity, R ij Indicates disease and disease The strength of the association between them comes from the edge weight in the knowledge graph, α j For each disease node The importance weight of a is set according to its centrality or frequency of occurrence in the knowledge graph.

[0043] Furthermore, in step S3, the preliminary diagnosis result is output.

[0044] According to the inferred disease matching probability Sort the diseases and output the most likely rare disease diagnosis results. The disease list is based on Sort from high to low and take the first k top Results, sorting formula:

[0045] in, is the first k after sorting top List of diseases, Indicates based on Rank the diseases from most severe to least severe.

[0046] Furthermore, in step S4, the dialogue management and patient personalization methods are as follows:

[0047] Based on the preliminary diagnosis results of rare diseases in step S3, diseases with higher probabilities in the preliminary diagnosis results are prioritized for inquiry;

[0048] Dynamically adjust the depth of the questions based on the complexity and severity of different conditions.

[0049] The beneficial effects of the present invention are:

[0050] This invention constructs a rare disease knowledge graph. Even when training data is insufficient, it can still deduce the implicit connection between diseases and symptoms through the association relationships in the graph, greatly enhancing the ability to handle complex cases such as rare diseases. Through the graph reasoning mechanism, it can provide a more accurate list of rare diseases and reasonably prioritize the display of relevant diagnostic solutions to users, reducing the risk of misjudgment.

[0051] This invention uses a multimodal Transformer model to effectively integrate multi-source data such as medical records, medical images, and laboratory test results, generate a unified comprehensive feature representation, and deeply integrate the ability to comprehensively analyze the patient's health status from different perspectives, thereby improving the accuracy of diagnosis.

[0052] Based on the preliminary diagnosis results, the present invention can adjust the order of inquiries and the depth of questions according to the specific conditions of different patients. For situations with a high probability of rare diseases, the system will prioritize exploring key symptoms related to the disease, providing a more accurate and personalized inquiry experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 This is a flowchart of the deep learning-based medical intelligent dialogue method of the present invention. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0057] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0058] Example 1, with reference to Figure 1 This embodiment provides a medical intelligent dialogue method based on deep learning, including the following steps:

[0059] Step S1: construct a rare disease knowledge graph.

[0060] Based on existing literature, clinical guidelines, expert knowledge, and data from medical research, we structure information on the causes, symptoms, diagnostic methods, treatment options, and drug interactions of rare diseases to generate a knowledge graph. In the absence of sufficient data training, the knowledge graph serves as a knowledge base for preliminary diagnosis, providing a list of potential rare diseases associated with symptoms.

[0061] The knowledge graph contains nodes for symptoms, diseases, drugs, and diagnostic methods. The knowledge graph serves as the core knowledge source for the dialogue system.

[0062] In step S1, the method of constructing the rare disease knowledge graph is as follows:

[0063] Based on literature, clinical guidelines and expert knowledge, collect relevant data on rare diseases, including causes, symptoms, diagnostic methods, treatment options and drug interactions, and define the data matrix D∈R m×n , where: D = {d ij}, d ij Represents the jth category of information for the i-th rare disease, including causes, symptoms, and drugs. m is the total number of rare diseases, and n is the number of categories of information related to each disease. The heterogeneous data is normalized using the normalization function:

[0064] in, is the normalized data, μ j is the mean of the j-th type of information, σ j is the standard deviation of the j-th category information, for each rare disease Define the following nodes:

[0065] S i ={s1,s2,…,s p} indicates disease Related p symptoms,

[0066] Expression and disease Related q treatment methods,

[0067] M i ={m1,m2,…,m r} indicates disease Related r drugs,

[0068] The relationship between nodes is represented by the relationship matrix R∈R k×k express: Among them, R ij Indicates disease and diseases The strength of the relationship between , f(·) represents the relationship definition function based on similarity, using the Jaccard similarity coefficient;

[0069] Use graph embedding method to embed nodes and relationships in knowledge graph into vector space, let for each node v i , and its corresponding vector is represented as v i ∈R d , d is the embedding dimension, then the relationship r in the knowledge graph ij The embedding representation is: ij =v i +rv j , where r ij Indicates disease and diseases The relationship between them is represented by the embedding of r, v i and v j Diseases and The node vector representation of the embedded node v i And the relationship r, deduce the possibility of potential disease, and define the inference probability function: in, When the embedding representation S of a given symptom set is expressed, the disease The diagnostic probability of S is the embedded representation of the given symptom set. Based on the inference result, the most likely diagnosis result of the rare disease is output;

[0070] Diagnosis results The score is: in, Indicates disease With input symptoms k The matching degree, ω k Symptoms k The weight of the symptom is used to sort and output a potential list of rare diseases based on the matching degree between the comprehensive symptoms and the disease;

[0071] Specifically, the constructed rare disease knowledge graph achieves highly structured integration of medical knowledge. The knowledge graph covers information such as the causes, symptoms, diagnostic methods, treatment plans, and drug interactions of rare diseases. In the absence of sufficient training data, it can rely on existing medical knowledge for preliminary diagnosis. Based on the embedded similarity between symptoms and disease nodes, it can quickly identify a list of possible rare diseases.

[0072] Step S2, integrated multimodal data processing,

[0073] Multimodal data, including medical records, medical images, and laboratory test results, is introduced and processed using a multimodal Transformer model. Node information in the knowledge graph is used as prior knowledge and jointly encoded and analyzed with the input multimodal data to generate a comprehensive understanding of the multimodal information, including feature extraction results based on patient symptoms, images, and genetic data, namely multimodal features.

[0074] In step S2, a multimodal Transformer model is used to jointly encode and analyze different data types such as medical records, medical images, and laboratory test results:

[0075] Process medical record text data, extract symptoms, diagnosis and medical history information, and set the medical record text sequence as T 1 ={t1,t2,…,t n}, where t i The word vector embedding representing the i-th word is converted into a high-level semantic representation through the self-attention mechanism: in, Represents the high-dimensional semantic representation of medical record text, d is the embedding dimension, are the query, key, and value matrices of the text sequence, respectively, d k is the dimension of each attention head, T represents the transposition operation, and n is the length of the medical record text;

[0076] For medical image data, convolutional neural network CNN is used for feature extraction. The input medical image is a three-dimensional matrix I∈R h×w×c , where h and w are the height and width of the image, and c is the number of channels. Through the CNN network, it is converted into a low-dimensional feature representation: F I =CNN(I), where F I ∈R m×d represents the extracted image features, m is the number of spatial locations in the feature map, and d is the embedding dimension;

[0077] Laboratory test results are represented by a vector L = {l1,l2,…,l k}, where l i The result of the i-th detection is mapped to the feature space through the fully connected layer: F L =W L L+b L , where F L ∈R d Represents the characteristic representation of laboratory test results, W L ∈R d×k is the weight matrix, b L ∈R d is the bias term, k is the number of detection items, and d is the embedding dimension;

[0078] In step S2, a multimodal Transformer model is used, and the features of the three types of data are H T 、F I and F L , fused through a multimodal self-attention mechanism:

[0079] Where Z∈R (n+m+1)×d is the fused multimodal feature representation, where n, m, and 1 correspond to the number of features of text, image, and laboratory test data, respectively, and Q M ,K M ,V M ∈R (n+m+1)×d are the fused query, key, and value matrices respectively, and d is the embedding dimension, where:

[0080] Q M =[H T ; F I ; F L ]W Q , K M =[H T ; F I ; F L ]W K , V M =[H T ; F I ; F L ]W V , here

[0081] H T ; F I ; F L ] represents the concatenation of text, image and laboratory test features, W Q 、W K 、W V ∈R d×d The mapping matrices for query, key, and value respectively;

[0082] The fused multimodal feature Z is obtained through the multimodal self-attention mechanism, and global pooling is performed to generate the final comprehensive feature vector F M :F M =GlobalPool(Z), where F M ∈R d is a multimodal feature vector used for subsequent diagnostic reasoning and symptom matching. The multimodal feature vector F M As a result of comprehensive understanding of multimodal information; achieving feature extraction and comprehensive understanding of multimodal data;

[0083] Specifically, the introduction of multimodal data processing (including medical records, medical images, and laboratory test results) effectively enhances diagnostic accuracy. The multimodal Transformer model is used to fuse data from various sources and generate a unified comprehensive feature representation, comprehensively analyzing the patient's condition from multiple perspectives and improving the ability to identify complex cases.

[0084] Step S3, knowledge graph reasoning and symptom matching,

[0085] Use knowledge graphs for symptom matching and combine multimodal data for preliminary diagnosis;

[0086] In step S3, based on the knowledge graph generated in step S1, the patient's input symptoms and multimodal features are matched through the knowledge reasoning mechanism to output a preliminary diagnosis result, including a list of possible diseases, suspected causes, and preliminary treatment recommendations;

[0087] In step S3, the patient's input symptoms and multimodal features are matched with the knowledge graph generated in step S1 through the knowledge reasoning mechanism, and a preliminary diagnosis result is output:

[0088] Suppose rare disease knowledge graph G = (V, E), where V is a node set representing disease, symptom, drug, and diagnostic method entities, and E is an edge set representing the relationship between entities. For each node v i ∈V, using vector v i ∈R d Represents, where d is the embedding dimension, and the node embedding formula is: i =f(v i ), where v i is the node v in the knowledge graph i The embedding vector, f(v i ) is to put the node v i Functions that convert to embedding vectors, which are converted by graph embedding methods such as TransE or GraphSAGE;

[0089] Multimodal feature F M A comprehensive vector representing the patient's input symptoms, medical imaging, and laboratory test results, with the same dimension as the knowledge graph node embedding v i Same, multimodal feature F M Match with nodes in the knowledge graph, F M =GlobalPool(Z), where F M is the fused multimodal feature vector, representing the comprehensive information of the patient, and Z is the feature matrix after fusing text, image, and laboratory data;

[0090] In step S3, symptoms and diseases are matched.

[0091] In the knowledge graph, there is an edge between disease and symptoms. ij ∈E, the multimodal features F of the patient input M and the disease node v in the graph i Perform matching and calculate the similarity between symptoms and diseases. The matching formula is:

[0092] Among them, S(F M ,v i ) represents the patient’s multimodal features F M With disease node v i The cosine similarity of |F is used to evaluate the matching degree between the input symptoms and the disease. M | and |v i |F M and v i The vector norm of , is the inner product of two vectors;

[0093] Define a relationship-based reasoning model, deduce the relationships in the knowledge graph, and calculate the matching degree of potential diseases. The reasoning formula is: in, Represents the patient's multimodal features F M Matching diseases The probability of S(F M ,v j ) is the multimodal feature and disease The similarity, R ij Indicates disease and disease The strength of the association between them comes from the edge weight in the knowledge graph, α j For each disease node The importance weight of a word is set according to its centrality or frequency of occurrence in the knowledge graph;

[0094] In step S3, the preliminary diagnosis result is output.

[0095] According to the inferred disease matching probability Sort the diseases and output the most likely rare disease diagnosis results. The disease list is based on Sort from high to low and take the first k top Results, sorting formula:

[0096] in, is the first k after sorting top List of diseases, Indicates based on Rank the diseases from most severe to least severe;

[0097] Step S4: Dialogue management and patient personalization.

[0098] Dynamically adjust the order and content of inquiries based on the diagnosis results obtained in step S3, and adjust the depth and direction of questions for patients with different conditions;

[0099] In step S4, the dialogue management and patient personalization methods are as follows:

[0100] Based on the preliminary diagnosis results of rare diseases in step S3, diseases with higher probabilities in the preliminary diagnosis results are prioritized for inquiry;

[0101] Dynamically adjust the depth of questions based on the complexity and severity of different conditions;

[0102] Specifically, by combining knowledge graphs and multimodal features, and using reasoning mechanisms to achieve deeper diagnostic capabilities, it not only relies on simple matching between symptoms and diseases, but also deduces through the association relationships in the graph. It can identify implicit associations between symptoms and potential rare diseases, greatly improving the depth and breadth of diagnosis.

[0103] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A medical intelligent dialogue method based on deep learning, characterized by: include, Step S1: construct a rare disease knowledge graph. Based on existing literature, clinical guidelines, expert knowledge, and data from medical research, the causes, symptoms, diagnostic methods, treatment options, and drug interactions of rare diseases are structured to generate a knowledge graph. Step S2: Integrated multimodal data processing. Multimodal data, including medical records, medical images, and laboratory test results, is introduced and processed using a multimodal Transformer model. Node information in the knowledge graph is used as prior knowledge and jointly encoded and analyzed with the input multimodal data to generate a comprehensive understanding of the multimodal information, including feature extraction results based on patient symptoms, images, and genetic data, namely multimodal features. Step S3: knowledge graph reasoning and symptom matching, using the knowledge graph to perform symptom matching and combining multimodal data for preliminary diagnosis; Step S4: Dialogue management and patient personalization. Based on the diagnosis results obtained in step S3, the order and content of the inquiries are dynamically adjusted, and the depth and direction of the questions are adjusted for patients with different conditions. In step S1, the method of constructing the rare disease knowledge graph is as follows: Based on literature, clinical guidelines and expert knowledge, collect relevant data on rare diseases, including causes, symptoms, diagnostic methods, treatment options and drug interactions, and define the data matrix D∈R m×n , where: D = {d ij }, d ij Represents the jth category of information for the i-th rare disease, including causes, symptoms, and drugs. m is the total number of rare diseases, and n is the number of categories of information related to each disease. The heterogeneous data is normalized using the normalization function: in, is the normalized data, μ j is the mean of the j-th type of information, σ j is the standard deviation of the j-th category information, for each rare disease Define the following nodes: S i ={s1,s2,…,s p } indicates disease Related p symptoms, Expression and disease Related q treatment methods, M i ={m1,m2,…,m r } indicates disease Related r drugs, The relationship between nodes is represented by the relationship matrix R∈R k×k express: Among them, R ij Indicates disease and diseases The strength of the relationship between , f(·) represents the relationship definition function based on similarity, using the Jaccard similarity coefficient; Use graph embedding method to embed nodes and relationships in knowledge graph into vector space, let for each node v i , and its corresponding vector is represented as v i ∈R d , d is the embedding dimension, then the relationship r in the knowledge graph ij The embedding representation is: ij =v i +rv j , where r ij Indicates disease and diseases The relationship between them is represented by the embedding of r, v i and v j Diseases and The node vector representation of the embedded node v i And the relationship r, deduce the possibility of potential disease, and define the inference probability function: in, When the embedding representation S of a given symptom set is expressed, the disease The diagnostic probability of S is the embedded representation of the given symptom set. Based on the inference result, the most likely diagnosis result of the rare disease is output; Diagnosis results The score is: in, Indicates disease With input symptoms k The matching degree, ω k Symptoms k The weight of the symptom is used to sort and output a potential list of rare diseases based on the matching degree between the comprehensive symptoms and the disease; In step S2, a multimodal Transformer model is used, and the features of the three types of data are H T 、F I and F L , fused through a multimodal self-attention mechanism: Where Z∈R (n+m+1)×d is the fused multimodal feature representation, where n, m, and 1 correspond to the number of features of text, image, and laboratory test data, respectively, and Q M ,K M ,V M ∈R (n+m+1)×d are the fused query, key, and value matrices respectively, and d is the embedding dimension, where: Q M =[H T ; F I ; F L ]W Q , K M =[H T ; F I ; F L ]W K , V M =[H T ; F I ; F L ]W V , where [H T ; F I ; F L ] represents the concatenation of text, image and laboratory test features, W Q 、W K 、W V ∈R d×d The mapping matrices for query, key, and value respectively; The fused multimodal feature Z is obtained through the multimodal self-attention mechanism, and global pooling is performed to generate the final comprehensive feature vector F M :F M =GlobalPool(Z), where F M ∈R d is a multimodal feature vector used for subsequent diagnostic reasoning and symptom matching. The multimodal feature vector F M As a result of comprehensive understanding of multimodal information; In step S3, symptoms and diseases are matched. In the knowledge graph, there is an edge between disease and symptoms. ij ∈E, the multimodal features F of the patient input M and the disease node v in the graph i Perform matching and calculate the similarity between symptoms and diseases. The matching formula is: Among them, S(F M ,v i ) represents the patient’s multimodal features F M With disease node v i The cosine similarity of |F is used to evaluate the matching degree between the input symptoms and the disease. M | and |v i |F M and v i The vector norm of , is the inner product of two vectors; Define a relationship-based reasoning model, deduce the relationships in the knowledge graph, and calculate the matching degree of potential diseases. The reasoning formula is: in, Represents the patient's multimodal features F M Matching diseases The probability of S(F M ,v j ) is the multimodal feature and disease The similarity, R ij Indicates disease and disease The strength of the association between them comes from the edge weight in the knowledge graph, α j For each disease node The importance weight of a word is set according to its centrality or frequency of occurrence in the knowledge graph; In step S4, the dialogue management and patient personalization methods are as follows: Based on the preliminary diagnosis results of rare diseases in step S3, diseases with higher probabilities in the preliminary diagnosis results are prioritized for inquiry; Dynamically adjust the depth of the questions based on the complexity and severity of different conditions.

2. A medical intelligent dialogue method based on deep learning according to claim 1, characterized in that: The knowledge graph contains nodes for symptoms, diseases, drugs, and diagnostic methods.

3. A medical intelligent dialogue method based on deep learning according to claim 2, characterized in that: In step S2, a multimodal Transformer model is used for joint encoding and analysis: Process medical record text data, extract symptoms, diagnosis and medical history information, and set the medical record text sequence as T 1 ={t1,t2,…,t n }, where t i The word vector embedding representing the i-th word is converted into a high-level semantic representation through the self-attention mechanism: in, Represents the high-dimensional semantic representation of medical record text, d is the embedding dimension, are the query, key, and value matrices of the text sequence, respectively, d k is the dimension of each attention head, T represents the transposition operation, and n is the length of the medical record text; For medical image data, convolutional neural network CNN is used for feature extraction. The input medical image is a three-dimensional matrix I∈R h×w×c , where h and w are the height and width of the image, and c is the number of channels. Through the CNN network, it is converted into a low-dimensional feature representation: F I =CNN(I), where F I ∈R m×d represents the extracted image features, m is the number of spatial locations in the feature map, and d is the embedding dimension; Laboratory test results are represented by a vector L = {l1,l2,…,l k }, where l i The result of the i-th detection is mapped to the feature space through the fully connected layer: F L =W L L+b L , where F L ∈R d Represents the characteristic representation of laboratory test results, W L ∈R d×k is the weight matrix, b L ∈R d is the bias term, k is the number of detection items, and d is the embedding dimension.

4. A medical intelligent dialogue method based on deep learning according to claim 3, characterized in that: In step S3, based on the knowledge graph generated in step S1, the symptoms input by the patient and the multimodal features are matched through the knowledge reasoning mechanism to output a preliminary diagnosis result, including a list of possible diseases, etiology speculation, and preliminary treatment recommendations.

5. A medical intelligent dialogue method based on deep learning according to claim 4, characterized in that: In step S3, the patient's input symptoms and multimodal features are matched with the knowledge graph generated in step S1 through the knowledge reasoning mechanism, and a preliminary diagnosis result is output: Suppose rare disease knowledge graph G = (V, E), where V is a node set representing disease, symptom, drug, and diagnostic method entities, and E is an edge set representing the relationship between entities. For each node v i ∈V, using vector v i ∈R d Represents, where d is the embedding dimension, and the node embedding formula is: i =f(v i ), where v i is the node v in the knowledge graph i The embedding vector, f(v i ) is to put the node v i Function to convert to embedding vector; Multimodal feature F M A comprehensive vector representing the patient's input symptoms, medical imaging, and laboratory test results, with the same dimension as the knowledge graph node embedding v i Same, multimodal feature F M Match with nodes in the knowledge graph, F M =GlobalPool(Z), where F M is the fused multimodal feature vector, representing the comprehensive information of the patient, and Z is the feature matrix after fusing text, image, and laboratory data.

6. The medical intelligent dialogue method based on deep learning according to claim 5, characterized in that: In step S3, the preliminary diagnosis result is output. According to the inferred disease matching probability Sort the diseases and output the most likely rare disease diagnosis results. The disease list is based on Sort from high to low and take the first k top Results, sorting formula: in, is the first k after sorting top List of diseases, Indicates based on Rank the diseases from most severe to least severe.

Images