Multi-mode traditional Chinese medicine diabetes syndrome type prediction method based on syndrome quantification

Abstract

The invention discloses a multi-mode traditional Chinese medicine diabetes syndrome type prediction method based on syndrome quantification, and relates to the problems that in traditional Chinese medicine syndrome differentiation treatment decision, diabetes syndrome type judgment is inaccurate, individualized syndrome differentiation treatment is difficult to achieve, and targeted prevention and treatment of diabetes are affected. Diabetes is a common metabolic disease, and in traditional Chinese medicine clinical practice, differentiated treatment schemes can be provided for patients through syndrome differentiation and typing. However, existing syndrome type judgment mainly depends on experience judgment of traditional Chinese medicine experts, is high in subjectivity and lacks a unified quantitative standard, so that syndrome type judgment results are unstable and inconsistent, and the application value of traditional Chinese medicine syndrome differentiation treatment in diabetes prevention and treatment is affected. Meanwhile, existing research mostly depends on single-mode information, multi-dimensional features such as tongue condition, pulse condition and inquiry are not effectively fused, and accuracy and interpretability of syndrome type prediction are further restricted. In order to solve the problem, the invention provides a multi-mode traditional Chinese medicine diabetes syndrome type prediction method based on syndrome quantification. Experiments show that the method has the following advantages: (1) objectivity and consistency of diabetes syndrome type judgment are improved through syndrome quantification; and (2) accuracy and interpretability of syndrome type prediction are remarkably improved through multi-modal fusion of tongue condition, pulse condition and inquiry. The method can be used for predicting the symptom type of the diabetic patient.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing, in particular to a multi-modal traditional Chinese medicine diabetes syndrome type prediction method based on syndrome quantification. BACKGROUND

[0002] Diabetes is a common metabolic disease, and long-term development can cause serious complications such as retinopathy, cardiovascular and cerebrovascular diseases, and kidney damage. Diabetic patients in traditional Chinese medicine clinical usually need to be treated according to the syndrome type to achieve individualized and targeted treatment. However, the existing diabetes syndrome type determination mainly relies on the clinical experience of traditional Chinese medicine experts, which has strong subjectivity and lacks unified quantitative standard, resulting in inconsistency between different doctors' diagnosis results, thereby affecting the accuracy and standardized application of syndrome differentiation and treatment.

[0003] Tongue and pulse, as an important part of the four diagnostic methods in traditional Chinese medicine, can reflect the overall metabolic level and the state of qi and blood yin and yang, and is an important basis for the differentiation of diabetes. However, in actual diagnosis and treatment, the acquisition and interpretation of tongue and pulse are still mainly based on manual observation and palpation, lacking objective and standardized quantitative tools. At the same time, although the clinical inquiry information of patients can supplement the understanding of symptoms, it has not been systematically integrated with tongue and pulse in existing research, and a multi-modal syndrome representation has not been formed. Therefore, how to quantitatively represent tongue image, pulse signal and clinical inquiry information, and combine deep learning method to realize multi-modal fusion, establish a unified diabetes syndrome type prediction model to improve the accuracy of diabetes syndrome type determination, and assist the syndrome differentiation and treatment of diabetes, has become an important direction of traditional Chinese medicine informatization and intelligent diagnosis and treatment research. SUMMARY

[0004] The purpose of the present application is to solve the problem of inaccurate diabetes syndrome type judgment in the prior art, which affects the effective syndrome differentiation and treatment of diabetes, and to provide a multi-modal traditional Chinese medicine diabetes syndrome type prediction method based on syndrome quantification.

[0005] The above invention purpose is mainly realized by the following technical scheme:

[0006] S1, the diabetic patients are included in the data set, the tongue image and pulse signal of the patients are collected, the basic symptoms of the patients are recorded by the traditional Chinese medicine experts through inquiry, and the diabetes syndrome type determined by the experts is used as the label to construct the diabetes syndrome type prediction data set, and the steps are:

[0007] (1) the diabetic patients are included in the data set, the tongue image and pulse signal of the patients are collected, the basic symptoms of the patients are recorded by the traditional Chinese medicine experts through inquiry, and the diabetes syndrome type determined by the experts is used as the label to construct the diabetes syndrome type prediction data set, and the steps are:

[0008] (2) According to the collected tongue and pulse conditions and interrogation data, a TCM expert determines the patient's diabetes syndrome type, and divides the patient into one of the following types: heat-dampness damaging fluid, deficiency of both qi and yin, dampness-heat accumulation, phlegm turbidity obstruction, deficiency of liver and kidney yin, or deficiency of both yin and yang;

[0009] (3) Combine the tongue image, pulse signal, and interrogation data with the corresponding syndrome type results to construct a diabetes syndrome type prediction dataset.

[0010] S2, analyze the features of the collected tongue image and pulse signal, calculate the quantified indexes of tongue color, tongue size, pulse rate, and wave amplitude, and obtain the syndrome quantification results of tongue and pulse.

[0011] (1) Perform color correction on the collected tongue image, segment the tongue region using a semantic segmentation model, and extract tongue features including tongue color, tongue size, tongue fur thickness, fur color distribution, tongue texture, and tongue crack.

[0012] (2) Perform filtering and normalization preprocessing on the collected pulse signal, extract time domain features such as pulse rate, wave amplitude, period, and wave area, perform frequency domain conversion on the pulse signal, and extract frequency domain features such as fundamental frequency, spectral energy distribution, spectral entropy, and spectral centroid.

[0013] S3, construct a multi-modal fusion diabetes syndrome type prediction model, which includes a tongue feature extractor, a pulse time domain feature extractor, a pulse frequency domain feature extractor, an interrogation data feature extractor, and a diabetes syndrome type predictor.

[0014] (1) Input the tongue image into the tongue feature extractor to extract tongue deep features, and perform feature fusion on the tongue deep features and tongue syndrome quantification results to obtain fused tongue features, as shown in formula (1):

[0015] (1)

[0016] wherein represents the fused tongue features, represents the tongue image, represents the tongue feature extractor, represents the tongue syndrome quantification results, represents feature fusion.

[0017] (2) input the pulse condition signal into a pulse condition time domain feature extractor to obtain a pulse condition time domain feature, perform frequency domain conversion on the pulse condition signal and input the pulse condition signal into a pulse condition frequency domain feature extractor to obtain a pulse condition frequency domain feature, perform feature fusion on the pulse condition time domain feature, the pulse condition frequency domain feature and a pulse condition syndrome quantization result to obtain fused tongue condition features, and the process is shown in formula (2):

[0018] (2)

[0019] wherein represents the fused pulse condition feature, represents the pulse condition signal, represents the pulse condition time domain feature extractor, represents the pulse condition frequency domain feature extractor, F represents the frequency domain conversion, represents the pulse condition syndrome quantization result, represents feature fusion.

[0020] (3) input the patient's inquiry data into an inquiry data feature extractor to obtain inquiry features, and the process is shown in formula (3):

[0021] (3)

[0022] wherein represents the inquiry features, represents the inquiry data, represents the inquiry data feature extractor.

[0023] (4) fuse the fused tongue condition features, the fused pulse condition features and the inquiry features and input into a diabetes syndrome type predictor to obtain a diabetes syndrome type prediction result, and the process is shown in formula (4):

[0024] (4)

[0025] wherein y represents the diabetes syndrome type prediction result, represents the diabetes syndrome type predictor, represents the fused tongue condition features, represents the fused pulse condition features, represents the inquiry features, represents feature fusion.

[0026] S4, the multi-modal fused diabetes syndrome type prediction model is supervised training with tongue condition images, pulse condition signals, inquiry data, tongue condition and pulse condition syndrome quantization results and corresponding diabetes syndrome type labels as training samples, the model parameters are updated through back propagation until the model converges, and the converged model is applied to diabetes syndrome type prediction.

[0027] Inventive effects

[0028] The application provides a multi-modal traditional Chinese medicine diabetes syndrome type prediction method based on syndrome quantification. The method first collects tongue image, pulse signal and inquiry clinical information of a diabetes patient, and determines the diabetes syndrome type by a traditional Chinese medicine expert to construct a diabetes syndrome type prediction data set; then performs feature analysis on the tongue image and the pulse signal, calculates objective quantification indexes such as tongue color, tongue fur thickness, pulse rate and wave amplitude to form syndrome quantification results of the tongue and the pulse; further constructs a multi-modal fusion diabetes syndrome type prediction model, which includes a tongue feature extractor, a pulse time domain feature extractor, a pulse frequency domain feature extractor, an inquiry data feature extractor and a diabetes syndrome type predictor; finally, the model is trained to convergence by using the multi-modal data of the patient and the corresponding diabetes syndrome type label, and the converged model is used for diabetes syndrome type prediction to output diabetes syndrome type prediction results such as heat-dampness damaging fluid syndrome, qi-yin deficiency syndrome, damp-heat accumulation syndrome, phlegm turbidity obstruction syndrome, liver-kidney yin deficiency syndrome and yin-yang deficiency syndrome. Experiments show that the method has the following advantages: (1) the objective expression of the tongue and the pulse is realized through syndrome quantification, and the accuracy and consistency of the diabetes syndrome type determination are improved; (2) the multi-modal fusion of the tongue, the pulse and the inquiry information enhances the feature representation ability and the interpretability of the model; (3) the method provides an auxiliary tool for the treatment of diabetes, and helps to improve the targeted prevention and treatment level of diabetes. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 A multi-modal traditional Chinese medicine diabetes syndrome type prediction method based on syndrome quantification in the embodiment of the application is shown in the flowchart.

[0030] Figure 2 A tongue syndrome quantification flowchart in the embodiment of the application is shown in the flowchart.

[0031] Figure 3 A pulse syndrome quantification flowchart in the embodiment of the application is shown in the flowchart.

[0032] Figure 4 A multi-modal fusion diabetes syndrome type prediction model structure diagram in the embodiment of the application is shown in the structure diagram. DETAILED DESCRIPTION DETAILED DESCRIPTION

[0034] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are some embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0035] As Figure 1As shown, the multi-modal TCM diabetes syndrome type prediction method based on syndrome quantification comprises the following steps:

[0036] S1, the diabetic patients are included in the data set, the tongue image is obtained by the standardized tongue image acquisition instrument under the unified illumination condition, the pulse wave signal is collected by the electronic pulse diagnosis instrument at the radial artery part, the basic symptom information including thirst, diet change, abnormal defecation, sleep quality, fatigue, etc. is collected by the TCM expert through inquiry, and the structured inquiry record is formed, the tongue image, pulse image and inquiry data are combined, and the TCM expert determines the diabetes syndrome type;

[0037] S2, the tongue image and pulse signal are analyzed, the tongue color, tongue size, pulse rate, wave amplitude and other quantitative indexes are calculated, and the syndrome quantification results of the tongue image and pulse image are obtained;

[0038] S3, a multi-modal fusion diabetes syndrome type prediction model is constructed, the model comprises a tongue image feature extractor, a pulse image time domain feature extractor, a pulse image frequency domain feature extractor, an inquiry data feature extractor and a diabetes syndrome type predictor, the tongue image feature extractor and the pulse image frequency domain feature extractor take ResNet-50 as the backbone network, the pulse image time domain feature extractor adopts one-dimensional Transformer based on time sequence as the backbone network, the inquiry data feature extractor adopts the encoder structure based on Transformer, and the diabetes syndrome type predictor takes the full connection classification network as the backbone structure;

[0039] S4, the multi-modal fusion diabetes syndrome type prediction model is used as the training sample of the tongue image, pulse signal, inquiry data, tongue image and pulse image syndrome quantification result and corresponding diabetes syndrome type label, in the training stage, the cross entropy loss function is used to calculate the loss, the model parameters are updated through back propagation, until the performance index on the validation set is no longer significantly improved, so as to determine the model convergence, the parameters of the converged model are fixed, and the model is used for diabetes syndrome type prediction.

[0040] The embodiments of the present application will be described in detail as follows:

[0041] The embodiments of the present application are specifically implemented as follows.

[0042] S1, as Figure 1As shown, the diabetic patients are included in the data set, the tongue is obtained by standardized tongue image acquisition instrument under uniform illumination conditions, the pulse is collected by electronic pulse diagnosis instrument in radial artery part, the patient is inquired by TCM expert, the basic symptom information including thirst, diet change, abnormal defecation, sleep quality, fatigue, etc. is collected, and the structured inquiry record is formed, combined with tongue and pulse and inquiry data, the patient is determined by TCM expert, the steps are as follows:

[0043] (1) The diabetic patients are included in the research data set according to the diagnostic criteria, the high-resolution tongue image is obtained by standardized tongue image acquisition instrument under uniform illumination conditions, the details such as tongue and coating are clear and visible, the pulse wave signal is collected by electronic pulse diagnosis instrument in radial artery part, the patient is systematically inquired by TCM expert, the inquiry content includes cold and heat, sweating, pain site and nature, head, chest and abdomen discomfort, ear and eye, sleep, diet and taste, defecation and female patient's menstrual condition, etc., and the structured inquiry record is formed;

[0044] (2) According to the collected tongue image, pulse signal and inquiry data, the patient is determined by TCM expert, and the patient is divided into one of six types, including heat and injury, deficiency of both qi and yin, damp and hot, phlegm and turbid, deficiency of liver and kidney yin, or deficiency of both yin and yang;

[0045] (3) The tongue image, pulse signal and inquiry data are combined with the corresponding syndrome type determination results to construct the diabetes syndrome type prediction data set.

[0046] S2, as shown in Figure 2 and Figure 3 , the collected tongue image and pulse signal are analyzed, the quantification indexes such as tongue color, tongue size, pulse rate and amplitude are calculated, and the syndrome quantification results of tongue and pulse are obtained, the steps are as follows:

[0047] (1) as shown in Figure 2 , the collected tongue image is color corrected to eliminate the influence of light and equipment difference, the semantic segmentation model Attention U-Net is used to segment the tongue region, the tongue and background region are effectively distinguished, the tongue related features including tongue color, tongue size, coating thickness, coating color distribution, tongue texture and tongue crack are extracted, and the tongue syndrome quantification results are formed by numerical method;

[0048] (2) as shown in Figure 3As shown, the collected pulse signals are filtered, normalized and preprocessed to remove noise and ensure that the signal amplitude is within a unified range. The pulse rate, amplitude, period and waveform area are extracted in the time domain. The pulse signal is converted to the frequency domain, and the Fourier transform is used to obtain the frequency spectrum representation. The fundamental frequency, spectral energy distribution, spectral entropy and spectral centroid are extracted in the frequency domain to form the syndrome quantification results of the pulse.

[0049] S3、as shown Figure 4 S3、as shown

[0050] (1) The tongue image is input into the tongue feature extractor to extract the tongue depth feature. The tongue depth feature and the tongue syndrome quantification result are fused to obtain the fused tongue feature, as shown in formula (1):

[0051] (1)

[0052] wherein represents the fused tongue feature, represents the tongue image, represents the tongue feature extractor, represents the tongue syndrome quantification result, represents the feature splicing operation;

[0053] (2) The pulse signal is input into the pulse time domain feature extractor to obtain the pulse time domain feature. The pulse signal is converted to the frequency domain and input into the pulse frequency domain feature extractor to obtain the pulse frequency domain feature. The pulse time domain feature, the pulse frequency domain feature and the pulse syndrome quantification result are fused to obtain the fused pulse feature, as shown in formula (2):

[0054] (2)

[0055] wherein represents the fused pulse feature, represents the pulse signal, represents the pulse time domain feature extractor, represents the pulse frequency domain feature extractor, F represents the Fourier transform, denotes the pulse syndrome quantitative result, denotes the feature splicing operation;

[0056] (3) input the patient's inquiry data into the inquiry data feature extractor to obtain inquiry features, and the process is shown in formula (3):

[0057] (3)

[0058] wherein denotes the inquiry features, denotes the inquiry data, denotes the inquiry data feature extractor;

[0059] (4) input the fused tongue features, the fused pulse features and the inquiry features into the diabetes syndrome type predictor to obtain the diabetes syndrome type prediction result, and the process is shown in formula (4):

[0060] (4)

[0061] wherein y denotes the diabetes syndrome type prediction result, denotes the diabetes syndrome type predictor, denotes the fused tongue features, denotes the fused pulse features, denotes the inquiry features, denotes the feature splicing operation.

[0062] S4, the multi-modal fusion diabetes syndrome type prediction model is used as a training sample with tongue image, pulse signal, inquiry data, tongue and pulse syndrome quantitative result and corresponding diabetes syndrome type label for supervised training, in the training stage, the cross entropy loss function is used to calculate the loss, the model parameters are iteratively updated through back propagation, until the performance index on the validation set is no longer significantly improved, so as to determine that the model converges, the parameters of the converged model are fixed, and the model is used for diabetes syndrome type prediction.

[0063] Table 1 is the diabetes syndrome type prediction performance comparison of the method of the present application and other methods, and Table 2 is the prediction performance comparison of the method of the present application on six diabetes syndrome types, which fully proves the excellent performance of the method of the present application in diabetes syndrome type prediction.

[0064] Table 1 is the diabetes syndrome type prediction performance comparison of the method of the present application and other methods, and Table 2 is the prediction performance comparison of the method of the present application on six diabetes syndrome types, which fully proves the excellent performance of the method of the present application in diabetes syndrome type prediction.

[0065] Method Accuracy AUC AUPR Tongue appearance single mode 0.618 0.668 0.652 Pulse condition single mode 0.627 0.674 0.66 Questionnaire data single mode 0.635 0.681 0.666 The method of the application 0.778 0.793 0.781

[0066] Table 2 is the prediction performance comparison of the method of the present application on different diabetes syndrome types

[0067] Diabetes syndrome type Accuracy AUC AUPR Heat-injurying fluid syndrome 0.785 0.801 0.790 Qi-yin deficiency syndrome 0.774 0.789 0.777 Damp-heat accumulation syndrome 0.763 0.778 0.767 Phlegm turbidity obstruction syndrome 0.755 0.771 0.760 Liver-kidney yin deficiency syndrome 0.742 0.758 0.747 Yin-yang deficiency syndrome 0.731 0.749 0.736

Claims

1. A method for predicting multi-modal traditional Chinese medicine (TCM) diabetes syndrome types based on syndrome quantification, characterized in that, Comprise the following steps: S1, the diabetic patients are included in the data set, the tongue image and the pulse signal of the patients are collected, the basic symptoms of the patients are recorded by the TCM experts through inquiry, and the diabetes syndrome type determined by the experts is combined as a label to construct a diabetes syndrome type prediction data set; S2, the collected tongue image and pulse signal are analyzed, the tongue color, tongue size, pulse rate, wave amplitude and other quantitative indexes are calculated, and the syndrome quantification results of the tongue and pulse are obtained; S3, a multi-modal fusion diabetes syndrome type prediction model is constructed, which includes a tongue feature extractor, a pulse time domain feature extractor, a pulse frequency domain feature extractor, an inquiry data feature extractor, and a diabetes syndrome type predictor; S4, the multi-modal fusion diabetes syndrome type prediction model is trained with the tongue image, pulse signal, inquiry data, tongue and pulse syndrome quantification results and corresponding diabetes syndrome type label as training samples, the model parameters are updated through back propagation until the model converges, and the converged model is applied to diabetes syndrome type prediction.

2. The multi-modal traditional Chinese medicine diabetes syndrome type prediction method based on syndrome quantification according to claim 1, characterized in that, In step S1, the diabetic patients are included in the data set, the tongue image and the pulse signal of the patients are collected, the basic symptoms of the patients are recorded by the TCM experts through inquiry, and the diabetes syndrome type determined by the experts is combined as a label to construct a diabetes syndrome type prediction data set, which comprises the following steps: S11, the diabetic patients are included in the data set, the tongue image and the pulse signal of the patients are collected, the basic symptoms of the patients are recorded by the TCM experts through inquiry, and the diabetes syndrome type determined by the experts is combined as a label to construct a diabetes syndrome type prediction data set, which comprises the following steps: S12, according to the collected tongue, pulse and inquiry data, the TCM experts determine the diabetes syndrome type of the patients, and divide the patients into one of the following types: heat-dryness syndrome, qi-yin deficiency syndrome, damp-heat accumulation syndrome, phlegm-turbid obstruction syndrome, liver-kidney yin deficiency syndrome or yin-yang deficiency syndrome; S13, the tongue image, pulse signal, inquiry data and corresponding syndrome type are combined to construct a diabetes syndrome type prediction data set.

3. The multi-modal traditional Chinese medicine diabetes syndrome type prediction method based on syndrome quantification according to claim 1, characterized in that, In step S2, the collected tongue image and pulse signal are analyzed, the tongue color, tongue size, pulse rate, wave amplitude and other quantitative indexes are calculated, and the syndrome quantification results of the tongue and pulse are obtained, which comprises the following steps: S21, the collected tongue image is color corrected, the tongue region is segmented by using a semantic segmentation model, and tongue features including tongue color, tongue size, tongue fur thickness, fur color distribution, tongue surface texture and tongue crack are extracted; S22, the collected pulse signal is filtered and normalized for pretreatment, time domain features such as pulse rate, wave amplitude, period and wave area are extracted, the pulse signal is converted into frequency domain, and frequency domain features such as fundamental frequency, spectrum energy distribution, spectrum entropy and spectrum center of gravity are extracted.

4. The multi-modal traditional Chinese medicine diabetes syndrome type prediction method based on syndrome quantification according to claim 1, characterized in that, In step S3, a multi-modal fusion diabetes syndrome type prediction model is constructed, which includes a tongue feature extractor, a pulse time domain feature extractor, a pulse frequency domain feature extractor, an inquiry data feature extractor, and a diabetes syndrome type predictor, which comprises the following steps: S31, input the tongue image into the tongue feature extractor to extract tongue depth features, and perform feature fusion on the tongue depth features and the tongue syndrome quantification result to obtain fused tongue features, which is shown in formula (1): (1) wherein represents a fused tongue feature, represents a tongue image, represents a tongue feature extractor, represents a tongue syndrome quantification result, represents feature fusion; S32, input the pulse signal into the pulse time domain feature extractor to obtain pulse time domain features, perform frequency domain conversion on the pulse signal, input the pulse signal into the pulse frequency domain feature extractor to obtain pulse frequency domain features, and perform feature fusion on the pulse time domain features, the pulse frequency domain features, and the pulse syndrome quantification result to obtain fused pulse features, which is shown in formula (2): (2) wherein represents a fused pulse condition feature, represents a pulse condition signal, represents a pulse condition time domain feature extractor, represents a pulse condition frequency domain feature extractor, F represents a frequency domain conversion, represents a pulse condition syndrome quantization result, represents a feature fusion; S33, input the patient's inquiry data into the inquiry data feature extractor to obtain inquiry features, which is shown in formula (3): (3) wherein represents an inquiry feature, represents inquiry data, represents an inquiry data feature extractor; S34, input the fused tongue features, the fused pulse features, and the inquiry features into the diabetes syndrome type predictor to obtain a diabetes syndrome type prediction result, which is shown in formula (4): (4) wherein y represents the diabetes syndrome type prediction result, represents a diabetes syndrome type predictor, represents a fused tongue feature, represents a fused pulse feature, represents an inquiry feature, represents feature fusion.

Images