Deep learning-based traditional Chinese medicine diagnosis question and answer method and system

Abstract

The invention discloses a traditional Chinese medicine diagnosis question and answer based on deep learning, and relates to the technical field of traditional Chinese medicine, and the method comprises the steps: remotely obtaining tongue condition information, pulse condition information, complexion diagnosis information and auscultation information of a patient; the color and state of the patient can be remotely observed; the disease condition of the patient is obtained through language communication; wherein the disease conditions of the patient comprise consciousness symptoms, the incidence process, the treatment course, the daily life, the plain physique, the past medical history and the family medical history; constructing a diagnosis and treatment knowledge graph by analyzing related traditional Chinese medicine literatures and medical case contents; wherein the medical case content comprises the disease condition, tongue condition information, pulse condition information, complexion diagnosis information and auscultation information of the patient; and in combination with syndrome type classification of a traditional Chinese medicine theory, analyzing the constructed disease condition knowledge graph by using a deep learning model so as to intelligently classify and diagnose the disease condition of the patient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of traditional Chinese medicine treatment, in particular to a traditional Chinese medicine diagnosis question and answer method and system based on deep learning. BACKGROUND

[0002] With the rapid development of information technology, traditional Chinese medicine has gradually combined with information technology. In the mode of remote medical diagnosis, traditional Chinese medicine "observation, smell, inquiry and palpation" is the characteristic diagnosis method of traditional Chinese medicine, and "inquiry" is an important link in clinical diagnosis of diseases. Traditional Chinese medicine inquiry is a method for doctors to understand the patient's condition by purposefully inquiring the patient or accompanying person, especially the patient's subjective feeling at that time. Through inquiry, a lot of information about the patient's physical condition can be obtained, and it plays an important role in four diagnoses.

[0003] Through thousands of years of medical practice, traditional Chinese medicine inquiry has gradually developed into a mature and standardized system. With the rapid development of computer artificial intelligence technology, the combination of modern technology and traditional Chinese medicine inquiry undoubtedly gives the traditional Chinese medicine inquiry system new connotations. How to realize the intelligentization of traditional Chinese medicine inquiry method by using modern scientific technology and make it widely used is an important topic.

[0004] At present, the existing inquiry process can make targeted questions for different departments or diseases, but it cannot make personalized questions according to the individual characteristics of patients and the different seasons of onset.

[0005] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0006] The embodiments of the present application provide a traditional Chinese medicine diagnosis question and answer method and system based on deep learning to solve the above technical problems.

[0007] The application provides a traditional Chinese medicine diagnosis and treatment method based on deep learning, comprising: remotely acquiring tongue information, pulse information, face color diagnosis information and smell diagnosis information of a patient; remotely observing the patient's appearance and manner; and acquiring the patient's disease condition through language communication; wherein the patient's disease condition includes subjective symptoms, disease process, treatment history, living habits, general constitution, past medical history and family medical history; by analyzing relevant traditional Chinese medicine literature and medical record content, a diagnosis and treatment knowledge graph is constructed; wherein the medical record content includes the patient's disease condition, tongue information, pulse information, face color diagnosis information and smell diagnosis information; combined with the syndrome type classification of traditional Chinese medicine theory, a deep learning model is used to analyze the constructed disease knowledge graph to intelligently classify and diagnose the patient's disease. The application provides a traditional Chinese medicine diagnosis and treatment system based on deep learning, comprising: a multi-modal data fusion module for remotely acquiring tongue information, pulse information, face color diagnosis information and smell diagnosis information of a patient; a remote medical interactive module enabling doctors to remotely observe the patient's appearance and manner through remote video; and acquiring the patient's disease condition through language communication; wherein the patient's disease condition includes subjective symptoms, disease process, treatment history, living habits, general constitution, past medical history and family medical history; a knowledge graph construction module for constructing a diagnosis and treatment knowledge graph by analyzing relevant traditional Chinese medicine literature and medical record content; wherein the medical record content includes the patient's disease condition, tongue information, pulse information, face color diagnosis information and smell diagnosis information; an intelligent diagnosis and decision module for combining the syndrome type classification of traditional Chinese medicine theory, using a deep learning model to analyze the constructed disease knowledge graph to intelligently classify and diagnose the patient's disease.

[0008] Further, the knowledge graph construction module constructs a diagnosis and treatment knowledge graph by analyzing relevant traditional Chinese medicine literature and medical record content, which is configured to: create a BERT-based traditional Chinese medicine entity recognition model to identify entities in the relevant traditional Chinese medicine literature and the medical record content; identify the relationships between the entities through a Transformer model; wherein the relationships between the entities include treatment relationships, symptom relationships and cause relationships; based on the relationships between the entities and the relationships between the attributes of the entities, use Neo4j to construct the diagnosis and treatment knowledge graph; wherein the nodes in the diagnosis and treatment knowledge graph represent the entities, and the edges in the diagnosis and treatment knowledge graph represent the relationships between the entities and the relationships between the attributes of the entities.

[0009] Furthermore, the entities in the relevant TCM literature and the medical case content include herbal entities, acupoint entities, literature disease entities, and patient condition entities; wherein, the attributes of the herbal entities include the herbal name and herbal attributes, and the herbal attributes include the properties, meridians, and efficacy; the attributes of the acupoint entities include the acupoint name, its associated meridian, location, and therapeutic functions; the attributes of the literature disease entities include the disease name, symptoms, etiology, and the corresponding TCM and Western medicine disease names; the attributes of the patient condition entities include the patient's disease condition, tongue appearance information, pulse appearance information, facial complexion diagnosis information, and auscultation and olfaction diagnosis information.

[0010] Furthermore, the intelligent diagnostic decision-making module, combining the syndrome classification of traditional Chinese medicine theory, uses a deep learning model to analyze the constructed disease knowledge graph to intelligently classify and diagnose the patient's condition. This module is configured as follows: A dynamic reasoning mechanism is created; wherein the dynamic reasoning mechanism is used to infer possible diseases and treatment plans based on the diagnostic knowledge graph; The theory of traditional Chinese medicine is transformed into theoretical rules, and these theoretical rules are integrated into the dynamic reasoning mechanism. By utilizing a capsule network model with dynamic routing and a dynamic reasoning mechanism that incorporates the aforementioned theoretical rules, the constructed disease knowledge graph is analyzed to intelligently classify and diagnose the patient's condition.

[0011] Furthermore, utilizing a capsule network model with dynamic routing and a dynamic reasoning mechanism incorporating the aforementioned theoretical rules, the constructed disease knowledge graph is analyzed to intelligently classify and diagnose the patient's condition, configured as follows: The entities, relationships between entities, and relationships between entity attributes in the diagnostic knowledge graph are used as basic data points. A model based on manifold learning theory is used to extract features from the basic data points to reduce the dimensionality of the basic data points and obtain basic features. The model based on manifold learning theory includes a locally linear embedding model or a t-SNE model. Construct the capsule network model for the dynamic routing, treating each basic feature as a capsule; wherein the output vector of the capsule contains the features of the entity; The dynamic routing algorithm learns the relationships between different capsules; the relationships between different capsules correspond to the relationships between the attributes of entities in the diagnostic knowledge graph; the dynamic routing algorithm is used to find the spatial hierarchical relationships and combination methods between different capsules; the dynamic routing algorithm can dynamically allocate connection weights between capsules at different levels, and determine the optimal connection method between lower-level capsules and higher-level capsules through repeated iterations. The connection weights between different capsules are determined based on the following formula; ; in, Basic features corresponding capsules and basic features Corresponding capsules Connection weights between them; It is a basic feature With basic features Distance function between; It is traversing the set Index of each candidate capsule; It is a predictive capsule A group of candidate capsules; It is a basic feature With basic features Distance function between; It is a learning parameter used to control the range of influence of distance; It is a prior knowledge score based on Traditional Chinese Medicine theory, which encodes basic features. and The prior probability of the relationship between them is determined based on the prior knowledge score, which is based on traditional Chinese medicine theory. Encodes basic features and The prior probability of the relationship between them; The relationships between different capsules output by the dynamic routing capsule network model are input into a dynamic reasoning mechanism that incorporates the theoretical rules, so as to intelligently classify and diagnose the patient's condition.

[0012] Furthermore, the multimodal data fusion module includes a tongue and facial color diagnosis information acquisition unit, a pulse diagnosis information acquisition unit, and an auscultation and olfaction information acquisition unit; wherein, The tongue and facial color diagnosis information acquisition unit is used to acquire video images of the patient's face and tongue by using a high-definition camera as a video acquisition device and combining it with a computer-controlled light source; using SlowFastNetworks and R-CNN, the patient's eye area, nose area, tongue area and cheek area are located and divided to extract the patient's facial color information, lip color information and mouth and nose dryness information. The pulse information acquisition unit is used to acquire the patient's pulse information using a pulse sensor assembly; the acquired pulse information is used to generate a pulse waveform and then denoise it; wherein, the pulse sensor assembly includes multiple pulse sensors, which correspond to the cun, guan, and chi positions in traditional Chinese medicine pulse diagnosis. The auscultation information acquisition unit is used to collect the patient's voice, breathing sounds and cough sounds using a microphone as an audio acquisition device; and to extract features from the collected voice, breathing sounds and cough sounds using Mel frequency cepstral coefficients and deep belief networks to obtain the patient's timbre features, pitch features and tone features.

[0013] Furthermore, the tongue and facial color diagnosis information acquisition unit utilizes SlowFastNetworks and R-CNN to locate and segment the patient's eye, nose, tongue, and cheek regions to extract the patient's facial color, lip color, and degree of dryness of the mouth and nose. This is configured as follows: The acquired video images of the patient's face and tongue were resized and normalized. Two parallel convolutional neural network (CNN) streams are constructed: a slow CNN stream for capturing static information in video images, and a fast CNN stream for capturing dynamic information in video images. The slow CNN stream uses a time step larger than a preset stride threshold to capture slow changes in video images of the face and tongue; the fast CNN stream uses a time step smaller than or equal to the preset stride threshold to capture fast movements in video images of the face and tongue. Static and dynamic features are extracted from video images of the face and tongue using two parallel convolutional neural network streams, respectively, and then fused into a feature map. The features extracted by the slow convolutional neural network stream are used to identify facial structure and color information, while the features extracted by the fast convolutional neural network stream are used to identify facial movements and dynamic changes. The fusion effect of the static and dynamic features is determined based on a feature fusion optimization index. ; in, The feature fusion optimization index is used to quantify the fusion effect of the static features and the dynamic features. It is in time Features extracted by the slow convolutional neural network stream; It is in time Features extracted by the fast convolutional neural network stream; It is the total time step; It is an exponential coefficient used to adjust the sensitivity to feature differences; β is the time step; β is a coefficient used to adjust the impact of the time step on the feature fusion optimization index. In the R-CNN framework, a region proposal network is used to generate candidate regions that may contain key parts in video images of the face and tongue; For the candidate regions generated by the region proposal network, feature extraction is performed on the feature map output by the fusion of two parallel convolutional neural network streams; Use a region-of-interest pooling layer to unify the features of candidate regions at different scales to a fixed size; Features of a uniform size are input into a classifier to classify each candidate region and determine whether each candidate region contains a specific key part. Use the bounding box regressor to adjust the position of the candidate region; Based on the key areas containing specific key parts in each adjusted position, the patient's facial color information, lip color information, and dryness information of the mouth and nose are extracted.

[0014] Furthermore, the auscultation information acquisition unit utilizes Mel-frequency cepstral coefficients and a deep belief network to extract features from the acquired speech, breathing, and cough sounds, obtaining the patient's timbre, pitch, and tone characteristics, which are configured as follows: The collected speech, breathing, and cough sounds are divided into frames of fixed length to obtain multi-frame audio signals. The multi-frame audio signal is pre-emphasized to enhance the energy of the high-frequency part in order to simulate the human ear's perception of high frequencies; Using a Mel filter bank, the linear frequencies of the multi-frame audio signals are converted into Mel frequencies to obtain a Mel frequency group, which simulates the human ear's perception of different frequencies. Perform a Fast Fourier Transform on each frame of audio signal and calculate the power spectrum of each frame of audio signal. Multiply the power spectrum of each frame of audio signal by the Mel frequency group to obtain the Mel spectrum of each frame of audio signal; The logarithm of the Mel spectrum is taken to simulate the nonlinear perceptual characteristics of the human ear; Construct a deep belief network, which contains multiple restricted Boltzmann machine layers, to learn high-level representations of audio features; The extracted log-Mel spectrum is used as input to train the deep belief network; Audio features at different levels are extracted through each layer of the deep belief network; The timbre features are extracted by utilizing the output of the deep belief network and combining it with the statistical properties of the Mel spectrum. By analyzing the periodic characteristics of each frame of audio signal and combining them with the statistical properties of the Mel spectrum, the pitch features and tone features are extracted; wherein, the periodic features include an autocorrelation function. ; ; in, It is the pitch feature mentioned above; It is the aforementioned tone feature; It is the autocorrelation function The delay; It is the frequency index of the Mel spectrum; and It is an adjustment coefficient used to control the contributions of the Mel spectrum and the autocorrelation function; It is the frequency index with the highest energy in the Mel spectrum; the autocorrelation function It is a measure of the signal and itself at different time delays. A function for calculating similarity; Indicated in the autocorrelation function In the middle, when time delay achieve At that time, the value of the autocorrelation function; It is obtained by passing the power spectrum of the audio signal through a Mel filter bank; each filter corresponds to a Mel frequency; This represents the Mel frequency value of the frequency with the highest energy among all Mel frequencies; It is the total energy of the Mel spectrum; It is the delay of the maximum value in the autocorrelation function; It is the total energy of the autocorrelation function; It is the total number of frequency indices of the Mel spectrum.

[0015] Furthermore, the deep learning-based traditional Chinese medicine diagnosis and answer system also includes: An electronic medical record archiving unit is used to store the patient's diagnostic results and treatment history; The information retrieval module is used to provide information retrieval services for the backend of the TCM diagnosis and answer system; wherein, the information retrieval module can support questions and answers about common diseases, commonly used drugs and health preservation methods, and provide intelligent question and answer services based on TCM diagnosis and treatment logic.

[0016] Based on the technical solution provided in this application, the system can perform personalized diagnosis according to the individual characteristics of patients and the different disease conditions obtained through verbal communication, making the consultation more tailored to the actual situation of the patient and improving the pertinence and effectiveness of the consultation; through the multimodal data fusion module, the system can remotely obtain the patient's tongue image information, pulse information, facial complexion information, and auscultation and olfaction information, realizing the modernization and remote access of the four diagnostic methods of traditional Chinese medicine ("inspection, auscultation and olfaction, inquiry and palpation"), improving the comprehensiveness and accuracy of diagnosis; the system enables doctors to observe the patient's expression and demeanor through remote video and obtain the patient's disease condition, including subjective symptoms and disease progression, through verbal communication, greatly facilitating communication between patients and doctors, especially for people with mobility difficulties or living in remote areas; the knowledge graph construction module constructs a diagnostic and treatment knowledge graph by analyzing relevant traditional Chinese medicine literature and medical records, enabling the system to utilize... By using historical data and TCM theory for deep learning and analysis, the system improves the scientific rigor and intelligence of diagnosis. The intelligent diagnostic decision-making module combines TCM syndrome classification with deep learning models to analyze the disease knowledge graph, enabling intelligent classification and diagnosis of patient conditions, thus improving efficiency and accuracy. This system explores a new TCM treatment model, integrating traditional TCM with modern technology. It not only provides intelligent decision support for TCM clinical diagnosis and treatment but also promotes the modernization of TCM diagnostic and treatment techniques. Through intelligent consultation and diagnosis processes, the system can quickly collect and analyze patient data, reducing repetitive work for doctors, improving treatment efficiency, and making TCM services more efficient. The system's construction and application of knowledge graphs contribute to the systematization and standardization of TCM knowledge, promoting its inheritance and innovation, and providing new tools and methods for TCM academic research and clinical practice. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 A flowchart of an optional deep learning-based question-and-answer method for traditional Chinese medicine diagnosis according to an embodiment of this application; Figure 2 This is a structural diagram of an optional deep learning-based traditional Chinese medicine diagnosis and answering system according to an embodiment of this application.

[0018] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, this application provides a deep learning-based question-and-answer method for traditional Chinese medicine diagnosis, including: S101, remotely acquires the patient's tongue image information, pulse image information, facial complexion diagnosis information, and auscultation diagnosis information; S102, remotely observe the patient's expression and demeanor; and obtain information about the patient's condition through verbal communication; the patient's condition includes subjective symptoms, disease progression, treatment process, daily life, usual physical condition, past medical history and family medical history; S103, by analyzing relevant TCM literature and medical records, construct a diagnostic and treatment knowledge graph; among which, the medical records include the patient's disease condition, tongue appearance information, pulse information, facial complexion information, and auscultation and olfaction information; S104 combines the syndrome classification of traditional Chinese medicine theory and uses a deep learning model to analyze the constructed disease knowledge graph in order to intelligently classify and diagnose the patient's condition.

[0021] like Figure 2 As shown, this application provides a deep learning-based traditional Chinese medicine diagnosis and answering system, including: The multimodal data fusion module 201 is used to remotely acquire patients' tongue image information, pulse image information, facial color diagnosis information, and auscultation and olfaction diagnosis information; The telemedicine interaction module 202 enables doctors to remotely observe patients' expressions and demeanor via video and obtain information about their illness through verbal communication. This information includes the patient's subjective symptoms, disease progression, treatment history, daily routine, general health, past medical history, and family medical history. The knowledge graph construction module 203 is used to construct a diagnostic and treatment knowledge graph by analyzing relevant TCM literature and medical records; among which, the medical records include the patient's disease condition, tongue appearance information, pulse appearance information, facial complexion diagnosis information, and auscultation diagnosis information; The intelligent diagnosis and decision-making module 204 is used to combine the syndrome classification of traditional Chinese medicine theory and use a deep learning model to analyze the constructed disease knowledge graph in order to intelligently classify and diagnose the patient's condition.

[0022] Based on the embodiments provided in this application, the system can perform personalized diagnoses according to the individual characteristics of patients and the different disease conditions obtained through verbal communication, making the consultation more tailored to the actual situation of the patient and improving the pertinence and effectiveness of the consultation; through the multimodal data fusion module, the system can remotely obtain the patient's tongue image information, pulse information, facial complexion information, and auscultation and olfaction information, realizing the modernization and remote access of the four diagnostic methods of traditional Chinese medicine ("inspection, auscultation and olfaction, inquiry, and palpation"), improving the comprehensiveness and accuracy of diagnosis; the system enables doctors to observe the patient's expression and demeanor through remote video and obtain the patient's disease condition, including subjective symptoms and disease progression, through verbal communication, greatly facilitating communication between patients and doctors, especially for people with mobility difficulties or living in remote areas; the knowledge graph construction module constructs a diagnostic and treatment knowledge graph by analyzing relevant traditional Chinese medicine literature and medical records, enabling the system to utilize By deeply learning and analyzing historical data and traditional Chinese medicine (TCM) theories, the system improves the scientific rigor and intelligence of diagnosis. The intelligent diagnostic decision-making module, combining TCM syndrome classification with deep learning models to analyze the disease knowledge graph, enables intelligent classification and diagnosis of patient conditions, improving efficiency and accuracy. This system explores a new TCM treatment model, integrating traditional TCM with modern technology. It not only provides intelligent decision support for TCM clinical diagnosis and treatment but also promotes the modernization of TCM diagnostic and treatment techniques. Through intelligent consultation and diagnosis processes, the system can quickly collect and analyze patient data, reducing repetitive work for doctors, improving treatment efficiency, and making TCM services more efficient. Furthermore, the system's knowledge graph construction and application contribute to the systematization and standardization of TCM knowledge, promoting its inheritance and innovation, and providing new tools and methods for TCM academic research and clinical practice.

[0023] Furthermore, the knowledge graph construction module constructs a diagnostic and treatment knowledge graph by analyzing relevant TCM literature and medical records, and is configured as follows: Create a BERT-based TCM entity recognition model to identify entities in relevant TCM literature and medical records; The Transformer model is used to identify relationships between entities; these relationships include treatment relationships, symptom relationships, and etiology relationships. Among them, the treatment relationship includes the treatment relationship between herbs, acupoints and diseases, such as herb A being used to treat disease B; the symptom relationship includes the association between diseases and symptoms, such as disease C manifesting as symptoms; and the etiology relationship includes the connection between diseases and etiologies, such as disease E being caused by etiology F.

[0024] Based on the relationships between entities and the relationships between entity attributes, a medical knowledge graph is constructed using Neo4j; where nodes in the medical knowledge graph represent entities, and edges in the medical knowledge graph represent the relationships between entities and the relationships between entity attributes.

[0025] Furthermore, the entities in relevant TCM literature and medical records include herbal entities, acupoint entities, disease entities, and patient condition entities. Among them, the attributes of herbal entities include the herb name and herb properties, including properties, meridians, and efficacy. The attributes of acupoint entities include the acupoint name, its associated meridian, location, and therapeutic functions. The attributes of disease entities include the disease name, symptoms, etiology, and corresponding TCM and Western medicine disease names. The attributes of patient condition entities include the patient's disease condition, tongue appearance information, pulse information, facial complexion information, and auscultation and olfaction information.

[0026] Furthermore, the intelligent diagnostic decision-making module, combining the syndrome classification of traditional Chinese medicine theory, utilizes a deep learning model to analyze the constructed disease knowledge graph to intelligently classify and diagnose the patient's condition. It is configured as follows: Create a dynamic reasoning mechanism; whereby the dynamic reasoning mechanism is used to infer possible diseases and treatment options based on the diagnostic knowledge graph; The theory of traditional Chinese medicine is transformed into theoretical rules, and these rules are integrated into a dynamic reasoning mechanism. Using a capsule network model with dynamic routing and a dynamic reasoning mechanism that incorporates theoretical rules, the constructed disease knowledge graph is analyzed to intelligently classify and diagnose patients' conditions.

[0027] Furthermore, utilizing a capsule network model with dynamic routing and a dynamic reasoning mechanism incorporating theoretical rules, the constructed disease knowledge graph is analyzed to intelligently classify and diagnose patients' conditions. This is configured as follows: The entities, relationships between entities, and relationships between entity attributes in the diagnostic knowledge graph are used as basic data points. A model based on manifold learning theory is used to extract features from the basic data points to reduce the dimensionality of the basic data points and obtain basic features. Among them, the model based on manifold learning theory includes the locally linear embedding model or the t-SNE model. This can be achieved by constructing a mapping function that maps high-dimensional entities and relationships to a low-dimensional space while preserving their key characteristics and intrinsic structure. Construct a capsule network model with dynamic routing, where each basic feature is treated as a capsule; the output vector of each capsule contains the features of the entity. The dynamic routing algorithm learns the relationships between different capsules; the relationships between different capsules correspond to the relationships between the attributes of entities in the diagnostic knowledge graph; the dynamic routing algorithm is used to find the spatial hierarchical relationships and combination methods between different capsules; the dynamic routing algorithm can dynamically allocate connection weights between capsules at different levels, and determine the optimal connection method between lower-level capsules and higher-level capsules through repeated iterations. The connection weights between different capsules are determined based on the following formula; ; in, Basic features Corresponding capsules and basic features Corresponding capsules Connection weights between them; It is a basic feature With basic features Distance function between; It is traversing the set Index of each candidate capsule; It is a predictive capsule A group of candidate capsules; It is a basic feature With basic features Distance function between; It is a learning parameter used to control the range of influence of distance; It is a prior knowledge score based on Traditional Chinese Medicine theory, which encodes basic features. and The prior probability of the relationship between them is determined based on the prior knowledge score, which is based on traditional Chinese medicine theory. Encodes basic features and The prior probability of the relationship between them; In capsule networks, the concept of a "prediction capsule" refers to a capsule in the higher layers of the network used to predict a specific attribute or category of the entity represented by the lower-level capsules. In the structure of a capsule network, the output of a lower-level capsule (e.g., possibly representing local features in an image) is used to predict the state of a higher-level capsule (e.g., possibly representing the object category in the entire image). Here, "prediction" refers to the lower-level capsule's inference about the potential states (such as existence, pose, etc.) of the higher-level capsule.

[0028] During dynamic routing, each lower-level capsule generates prediction vectors for all possible higher-level capsules. These prediction vectors represent the lower-level capsule's prediction of the higher-level capsule's state. Then, through a dynamic routing algorithm, the model iteratively adjusts the weights of these predictions to determine which lower-level capsule predictions should be sent to specific higher-level capsules. This process involves calculating the consistency between the prediction vectors and the current state of the higher-level capsules, as well as updating the routing weights to make the predictions more accurate.

[0029] Specifically, the concept of prediction capsules can be understood as follows: Prediction vector generation: Each lower-level capsule generates a prediction vector for each higher-level capsule. This vector contains the lower-level capsule's prediction of the higher-level capsule's pose and existence. Dynamic routing weights: The dynamic routing algorithm calculates routing weights from lower-level capsules to higher-level capsules. These weights reflect the strength of the lower-level capsule's prediction of the higher-level capsule's state. Iterative adjustment: Through iterative processes, the model continuously adjusts these routing weights to optimize the state of the higher-level capsules, making them better represent the entities in the input data. Consistency assessment: In each iteration, the model evaluates the consistency between the prediction vector of the lower-level capsule and the current state of the higher-level capsule. This is done by calculating the vector dot product; a higher dot product indicates stronger consistency. Final state determination: After multiple iterations, the model determines the final state of each higher-level capsule. This state reflects the existence probability and pose information of the entity represented by the higher-level capsule.

[0030] Therefore, "prediction capsules" here refer to the higher-level capsules in a capsule network, whose states are determined jointly by the prediction vectors of the lower-level capsules and the dynamic routing algorithm. This process enables capsule networks to capture the hierarchical structure and spatial relationships in the input data, thereby improving their ability to recognize entities.

[0031] This formula calculates the connection weights between capsules by combining a distance metric and prior knowledge of Traditional Chinese Medicine (TCM) theory. The distance metric reflects the similarity between two capsules in the feature space, while the prior knowledge function reflects the strength of the relationship between entities in TCM theory. In this way, we can consider not only data-level similarity but also the rationality at the TCM theoretical level during dynamic routing. Parameters This design allows the model to learn how to balance the influence of distance, enabling it to adapt to different data distributions and task requirements. This makes the dynamic routing algorithm more flexible and better able to capture the complex relationships within the TCM diagnostic knowledge graph. Through this formula, we can implement a more refined and theory-driven dynamic routing mechanism in capsule networks, thereby improving the accuracy and reliability of the TCM diagnostic question-answering system. The formula's design takes into account the unique characteristics of the TCM diagnostic knowledge graph, making it more suitable for practical application scenarios.

[0032] The relationships between different capsules output by the dynamically routed capsule network model are input into a dynamic reasoning mechanism that incorporates theoretical rules to intelligently classify and diagnose patients' conditions.

[0033] The dynamic routing algorithm refines the connections between capsules through an iterative process. In each iteration, the algorithm calculates the similarity between capsules and assigns weights through a softmax layer. These weights reflect the influence of one capsule on another. This process is repeated until convergence, meaning the connection weights between capsules stabilize. Finding spatial hierarchical relationships and combination patterns: The dynamic routing algorithm searches for spatial hierarchical relationships and combination patterns between different capsules, which helps the model understand the complex knowledge graph structure of traditional Chinese medicine diagnosis and treatment. For example, if a herbal capsule (CherbCherb) has a strong connection with a disease capsule (CdiseaseCdisease), it may mean that this herb is commonly used to treat that disease. Through the steps described above, the dynamically routed capsule network model can learn the complex relationships between entities and their attributes in a diagnostic knowledge graph, providing a basis for intelligent classification and diagnosis in Traditional Chinese Medicine (TCM). This method enables the model to capture subtle differences between entities and attributes in TCM diagnosis, thereby improving the accuracy and efficiency of diagnosis.

[0034] Furthermore, the multimodal data fusion module includes a tongue and facial color diagnosis information acquisition unit, a pulse diagnosis information acquisition unit, and an auscultation and olfaction diagnosis information acquisition unit; among them, The tongue and facial color diagnosis information acquisition unit uses a high-definition camera as a video acquisition device, combined with a computer-controlled light source, to acquire video images of the patient's face and tongue; using SlowFastNetworks and R-CNN, it locates and divides the patient's eye area, nose area, tongue area and cheek area to extract the patient's facial color information, lip color information and mouth and nose dryness information. The pulse information acquisition unit is used to acquire the patient's pulse information using the pulse sensor assembly; the acquired pulse information is used to generate a pulse waveform and then denoise it; the pulse sensor assembly includes multiple pulse sensors, which correspond to the cun, guan, and chi positions in traditional Chinese medicine pulse diagnosis. The auscultation information acquisition unit uses a microphone as an audio acquisition device to collect the patient's voice, breathing sounds, and cough sounds; it uses Mel frequency cepstral coefficients and deep belief networks to extract features from the collected voice, breathing sounds, and cough sounds to obtain the patient's timbre, pitch, and tone features.

[0035] Furthermore, the tongue and facial color diagnosis information acquisition unit utilizes SlowFastNetworks and R-CNN to locate and segment the patient's eye, nose, tongue, and cheek regions to extract facial color, lip color, and mouth and nose dryness information, which is configured as follows: The video images of the patient's face and tongue were resized and normalized. Two parallel convolutional neural network streams are constructed: a slow convolutional neural network stream for capturing static information in video images, and a fast convolutional neural network stream for capturing dynamic information in video images. The slow convolutional neural network stream uses a time step larger than a preset step size threshold to capture slow changes in video images of the face and tongue, such as subtle changes in facial expressions. Fast convolutional neural network streams capture fast movements in video images of the face and tongue, such as lip movements when speaking, using time steps less than or equal to a preset stride threshold; Static and dynamic features from video images of the face and tongue are extracted using two parallel convolutional neural network streams, respectively, and then fused into a feature map. The features extracted by the slow convolutional neural network stream are used to identify facial structure and color information, while the features extracted by the fast convolutional neural network stream are used to identify facial movements and dynamic changes. The fusion effect of static and dynamic features is determined based on feature fusion optimization metrics. ; in, This represents a feature fusion optimization index, used to quantify the fusion effect of static and dynamic features; It is in time Features extracted by a slow convolutional neural network stream; It is in time Features extracted by a fast convolutional neural network stream; It is the total time step; It is an exponential coefficient used to adjust the sensitivity to feature differences; β is the time step; β is a coefficient used to adjust the impact of the time step on the feature fusion optimization index. In the R-CNN framework, a region proposal network is used to generate candidate regions that may contain key parts in video images of the face and tongue; For candidate regions generated by the region proposal network, feature extraction is performed on the feature map output by the fusion of two parallel convolutional neural network streams; Use a region-of-interest pooling layer to unify the features of candidate regions at different scales to a fixed size; Features of a uniform size are input into a classifier to classify each candidate region and determine whether each candidate region contains a specific key part. Use the bounding box regressor to adjust the position of the candidate region; Based on the key areas containing specific key parts in each adjusted position, extract the patient's facial color information, lip color information, and information on the degree of dryness of the mouth and nose.

[0036] Furthermore, the auscultation information acquisition unit utilizes Mel-frequency cepstral coefficients and a deep belief network to extract features from the acquired speech, breathing, and cough sounds, obtaining the patient's timbre, pitch, and tone characteristics, which are configured as follows: The collected speech, breathing, and cough sounds are divided into frames of fixed length to obtain multi-frame audio signals. Pre-emphasis is applied to multiple frames of audio signals to enhance the energy of the high-frequency components, thereby simulating the human ear's perception of high frequencies; By using a Mel filter bank, the linear frequencies of multiple audio signals are converted into Mel frequencies to obtain a Mel frequency group, which can simulate the human ear's perception of different frequencies. Perform a Fast Fourier Transform on each frame of audio signal and calculate the power spectrum of each frame of audio signal. Multiply the power spectrum of each frame of audio signal by the Mel frequency group to obtain the Mel spectrum of each frame of audio signal; The logarithm of the Mel spectrum is taken to simulate the nonlinear perceptual characteristics of the human ear; Construct a deep belief network, which contains multiple restricted Boltzmann machine layers, to learn high-level representations of audio features; The extracted log-Mel spectrum is used as input to train a deep belief network; By extracting audio features at different levels through each layer of the deep belief network; By utilizing the output of a deep belief network and combining it with the statistical properties of the Mel spectrum, timbre features are extracted. By analyzing the periodic characteristics of each frame of audio signal and combining them with the statistical properties of the Mel spectrum, pitch and tone features are extracted; among them, the periodic features include the autocorrelation function. ; ; in, It is the pitch feature mentioned above; It is the aforementioned tone feature; It is an autocorrelation function The delay; It is the frequency index of the Mel spectrum; and It is an adjustment coefficient used to control the contribution of the Mel spectrum and the autocorrelation function; It is the frequency index with the highest energy in the Mel spectrum; autocorrelation function It is a measure of the signal and itself at different time delays. A function for calculating similarity; Indicates the autocorrelation function In the middle, when time delay achieve The value of the autocorrelation function at that time; It is obtained by passing the power spectrum of the audio signal through a Mel filter bank; each filter corresponds to a Mel frequency; This represents the Mel frequency value of the frequency with the highest energy among all Mel frequencies; It is the total energy of the Mel spectrum; It is the delay of the maximum value in the autocorrelation function; It is the total energy of the autocorrelation function; It is the total number of frequency indices in the Mel spectrum.

[0037] Furthermore, deep learning-based traditional Chinese medicine diagnosis and answer systems also include: Electronic medical record archiving unit is used to store patients' diagnostic results and treatment history; The information retrieval module provides information retrieval services for the backend of the TCM diagnosis and answer system. It supports questions and answers about common diseases, commonly used drugs, and health preservation methods, and provides intelligent question and answer services based on TCM diagnosis and treatment logic.

[0038] It should be noted that the embodiments implemented on the side of the deep learning-based TCM diagnosis and answer system in this application can be referenced with the embodiments implemented on the side of the deep learning-based TCM diagnosis and answer method, and will not be described in detail here.

[0039] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A deep learning-based question-and-answer method for traditional Chinese medicine diagnosis, characterized in that, include: Remotely obtain patients' tongue appearance information, pulse information, facial complexion information, and auscultation information; The patient's expression and demeanor were observed remotely. The patient's medical condition is obtained through verbal communication; the patient's medical condition includes subjective symptoms, disease progression, treatment process, daily life, general physical condition, past medical history, and family medical history. By analyzing relevant TCM literature and medical case records, a diagnostic and treatment knowledge graph is constructed; the medical case records include the patient's disease condition, tongue appearance information, pulse information, facial complexion information, and auscultation and olfaction information. By combining the syndrome differentiation of traditional Chinese medicine theory, a deep learning model is used to analyze the constructed disease knowledge graph in order to intelligently classify and diagnose the patient's condition.

2. A deep learning-based traditional Chinese medicine diagnosis and answering system, wherein the system implements the method as described in claim 1, characterized in that, include: The multimodal data fusion module is used to remotely acquire patients' tongue image information, pulse image information, facial color diagnosis information, and auscultation and olfaction diagnosis information; The telemedicine interaction module enables doctors to remotely observe the patient's expression and demeanor via video and to obtain information about the patient's condition through verbal communication. The patient's condition includes subjective symptoms, disease progression, treatment history, daily life, general health, past medical history, and family medical history. The knowledge graph construction module is used to construct a diagnostic and treatment knowledge graph by analyzing relevant TCM literature and medical case content; wherein, the medical case content includes the patient's disease condition, tongue appearance information, pulse appearance information, facial complexion diagnosis information, and auscultation diagnosis information; The intelligent diagnostic decision-making module combines syndrome classification based on traditional Chinese medicine theory and uses a deep learning model to analyze the constructed disease knowledge graph in order to intelligently classify and diagnose the patient's condition.

3. The deep learning-based traditional Chinese medicine diagnosis and answer system according to claim 2, characterized in that, The knowledge graph construction module constructs a diagnostic and treatment knowledge graph by analyzing relevant traditional Chinese medicine literature and medical records, and is configured as follows: A BERT-based TCM entity recognition model was created to identify entities in the relevant TCM literature and the medical case content. The Transformer model is used to identify relationships between the entities; these relationships include treatment relationships, symptom relationships, and etiology relationships. Based on the relationships between the entities and the relationships between the attributes of the entities, the diagnosis and treatment knowledge graph is constructed using Neo4j; wherein, the nodes in the diagnosis and treatment knowledge graph represent the entities, and the edges in the diagnosis and treatment knowledge graph represent the relationships between the entities and the relationships between the attributes of the entities.

4. The deep learning-based traditional Chinese medicine diagnosis and answer system according to claim 3, characterized in that, The entities in the relevant TCM literature and medical case content include herbal entities, acupoint entities, literature disease entities, and patient condition entities. The attributes of the herbal entities include the herb name and herb properties, including their nature, flavor, meridian tropism, and efficacy. The attributes of the acupoint entities include the acupoint name, its associated meridian, location, and therapeutic functions. The attributes of the literature disease entities include the disease name, symptoms, etiology, and their corresponding TCM and Western medicine names. The attributes of the patient condition entities include the patient's disease condition, tongue appearance information, pulse information, facial complexion information, and auscultation / olfaction information.

5. The deep learning-based traditional Chinese medicine diagnosis and answer system according to claim 3, characterized in that, The intelligent diagnostic decision-making module, combining the syndrome classification of traditional Chinese medicine theory, uses a deep learning model to analyze the constructed disease knowledge graph to intelligently classify and diagnose the patient's condition. It is configured as follows: A dynamic reasoning mechanism is created; wherein the dynamic reasoning mechanism is used to infer possible diseases and treatment plans based on the diagnostic knowledge graph; The theory of traditional Chinese medicine is transformed into theoretical rules, and these theoretical rules are integrated into the dynamic reasoning mechanism. By utilizing a capsule network model with dynamic routing and a dynamic reasoning mechanism that incorporates the aforementioned theoretical rules, the constructed disease knowledge graph is analyzed to intelligently classify and diagnose the patient's condition.

6. The deep learning-based traditional Chinese medicine diagnosis and answer system according to claim 5, characterized in that, The capsule network model utilizing dynamic routing and the dynamic reasoning mechanism incorporating the theoretical rules are used to analyze the constructed disease knowledge graph for intelligent classification and diagnosis of the patient's condition, configured as follows: The entities, relationships between entities, and relationships between entity attributes in the diagnostic knowledge graph are used as basic data points. A model based on manifold learning theory is used to extract features from the basic data points to reduce the dimensionality of the basic data points and obtain basic features. The model based on manifold learning theory includes a locally linear embedding model or a t-SNE model. Construct the capsule network model for the dynamic routing, treating each basic feature as a capsule; wherein the output vector of the capsule contains the features of the entity; The dynamic routing algorithm learns the relationships between different capsules; the relationships between different capsules correspond to the relationships between the attributes of entities in the diagnostic knowledge graph; the dynamic routing algorithm is used to find the spatial hierarchical relationships and combination methods between different capsules; the dynamic routing algorithm can dynamically allocate connection weights between capsules at different levels, and determine the optimal connection method between lower-level capsules and higher-level capsules through repeated iterations. The connection weights between different capsules are determined based on the following formula; ； in, Basic features Corresponding capsules and basic features Corresponding capsules Connection weights between them; It is a basic feature With basic features Distance function between; It is traversing the set Index of each candidate capsule; It is a predictive capsule A group of candidate capsules; It is a basic feature With basic features Distance function between; It is a learning parameter used to control the range of influence of distance; It is a prior knowledge score based on Traditional Chinese Medicine theory, which encodes basic features. and The prior probability of the relationship between them, and the prior knowledge score, are determined based on the aforementioned traditional Chinese medicine theory; Encodes basic features and The prior probability of the relationship between them; The relationships between different capsules output by the dynamic routing capsule network model are input into a dynamic reasoning mechanism that incorporates the theoretical rules, so as to intelligently classify and diagnose the patient's condition.

7. The deep learning-based traditional Chinese medicine diagnosis and answer system according to claim 2, characterized in that, The multimodal data fusion module includes a tongue and facial color diagnosis information acquisition unit, a pulse diagnosis information acquisition unit, and an auscultation and olfaction information acquisition unit; wherein... The tongue and facial color diagnosis information acquisition unit is used to acquire video images of the patient's face and tongue by using a high-definition camera as a video acquisition device and combining it with a computer-controlled light source; using SlowFastNetworks and R-CNN, the patient's eye area, nose area, tongue area and cheek area are located and divided to extract the patient's facial color information, lip color information and mouth and nose dryness information. The pulse information acquisition unit is used to acquire the patient's pulse information using a pulse sensor assembly; the acquired pulse information is used to generate a pulse waveform and then denoise it; wherein, the pulse sensor assembly includes multiple pulse sensors, which correspond to the cun, guan, and chi positions in traditional Chinese medicine pulse diagnosis. The auscultation information acquisition unit is used to collect the patient's voice, breathing sounds and cough sounds using a microphone as an audio acquisition device; and to extract features from the collected voice, breathing sounds and cough sounds using Mel frequency cepstral coefficients and deep belief networks to obtain the patient's timbre features, pitch features and tone features.

8. The deep learning-based traditional Chinese medicine diagnosis and answer system according to claim 7, characterized in that, The tongue and facial color diagnosis information acquisition unit utilizes SlowFastNetworks and R-CNN to locate and segment the patient's eye, nose, tongue, and cheek regions to extract facial color, lip color, and mouth and nose dryness information. It is configured as follows: The acquired video images of the patient's face and tongue were resized and normalized. Two parallel convolutional neural network (CNN) streams are constructed: a slow CNN stream for capturing static information in video images, and a fast CNN stream for capturing dynamic information in video images. The slow CNN stream uses a time step larger than a preset stride threshold to capture slow changes in video images of the face and tongue; the fast CNN stream uses a time step smaller than or equal to the preset stride threshold to capture fast movements in video images of the face and tongue. Static and dynamic features are extracted from video images of the face and tongue using two parallel convolutional neural network streams, respectively, and then fused into a feature map. The features extracted by the slow convolutional neural network stream are used to identify facial structure and color information, while the features extracted by the fast convolutional neural network stream are used to identify facial movements and dynamic changes. The fusion effect of the static and dynamic features is determined based on a feature fusion optimization index. ； in, The feature fusion optimization index is used to quantify the fusion effect of the static features and the dynamic features. It is in time Features extracted by the slow convolutional neural network stream; It is in time Features extracted by the fast convolutional neural network stream; It is the total time step; It is an exponential coefficient used to adjust the sensitivity to feature differences; β is the time step; β is a coefficient used to adjust the impact of the time step on the feature fusion optimization index. In the R-CNN framework, a region proposal network is used to generate candidate regions that may contain key parts in video images of the face and tongue; For the candidate regions generated by the region proposal network, feature extraction is performed on the feature map output by the fusion of two parallel convolutional neural network streams; Use a region-of-interest pooling layer to unify the features of candidate regions at different scales to a fixed size; Features of a uniform size are input into a classifier to classify each candidate region and determine whether each candidate region contains a specific key part. Use the bounding box regressor to adjust the position of the candidate region; Based on the key areas containing specific key parts in each adjusted position, the patient's facial color information, lip color information, and dryness information of the mouth and nose are extracted.

9. The deep learning-based traditional Chinese medicine diagnosis and answer system according to claim 7, characterized in that, The auscultation information acquisition unit utilizes Mel frequency cepstral coefficients and a deep belief network to extract features from the acquired speech, breathing, and cough sounds, obtaining the patient's timbre, pitch, and tone characteristics, which are configured as follows: The collected speech, breathing, and cough sounds are divided into frames of fixed length to obtain multi-frame audio signals. The multi-frame audio signal is pre-emphasized to enhance the energy of the high-frequency part in order to simulate the human ear's perception of high frequencies; Using a Mel filter bank, the linear frequencies of the multi-frame audio signals are converted into Mel frequencies to obtain a Mel frequency group, which simulates the human ear's perception of different frequencies. Perform a Fast Fourier Transform on each frame of audio signal and calculate the power spectrum of each frame of audio signal. Multiply the power spectrum of each frame of audio signal by the Mel frequency group to obtain the Mel spectrum of each frame of audio signal; The logarithm of the Mel spectrum is taken to simulate the nonlinear perceptual characteristics of the human ear; Construct a deep belief network, which contains multiple restricted Boltzmann machine layers, to learn high-level representations of audio features; The extracted log-Mel spectrum is used as input to train the deep belief network; Audio features at different levels are extracted through each layer of the deep belief network; The timbre features are extracted by utilizing the output of the deep belief network and combining it with the statistical properties of the Mel spectrum. By analyzing the periodic characteristics of each frame of audio signal and combining them with the statistical properties of the Mel spectrum, the pitch features and tone features are extracted; wherein, the periodic features include an autocorrelation function. ； ； in, It is the pitch feature mentioned above; It is the aforementioned tone feature; It is the autocorrelation function The delay; It is the frequency index of the Mel spectrum; and It is an adjustment coefficient used to control the contributions of the Mel spectrum and the autocorrelation function; It is the frequency index with the highest energy in the Mel spectrum; the autocorrelation function It is a measure of the signal and itself at different time delays. A function for calculating similarity; Indicated in the autocorrelation function In the middle, when time delay achieve At that time, the value of the autocorrelation function; It is obtained by passing the power spectrum of the audio signal through a Mel filter bank; each filter corresponds to a Mel frequency; This represents the Mel frequency value of the frequency with the highest energy among all Mel frequencies; It is the total energy of the Mel spectrum; It is the delay of the maximum value in the autocorrelation function; It is the total energy of the autocorrelation function; It is the total number of frequency indices of the Mel spectrum.

10. The deep learning-based traditional Chinese medicine diagnosis and answer system according to claim 2, characterized in that, The deep learning-based TCM diagnosis and question-answering system also includes: An electronic medical record archiving unit is used to store the patient's diagnostic results and treatment history; The information retrieval module is used to provide information retrieval services for the backend of the TCM diagnosis and answer system; wherein, the information retrieval module can support questions and answers about common diseases, commonly used drugs and health preservation methods, and provide intelligent question and answer services based on TCM diagnosis and treatment logic.

Images