An auxiliary identification system for children's TCM cold and heat syndromes based on a report generation model

Abstract

The present invention discloses a pediatric TCM cold and heat syndrome auxiliary identification system based on a report generation model. The system comprises a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor. The computer memory stores a trained report generation model and a multimodal classification model. The report generation model comprises an encoding portion and a decoding portion. When the computer processor executes the computer program, the following steps are performed: an image of the child's tongue surface is input into the trained report generation model; the image feature vector X obtained in the first step of the encoding portion is directly input into the decoding portion to generate a report, and the report is normalized into a key-value pattern; the key-valued clinical information, the medical consultation information, and the report generated by the report generation model are then input into the multimodal classification model to obtain a cold and heat syndrome classification result. The present invention has high interpretability for visual diagnosis images and improves the accuracy of cold and heat syndrome discrimination.

Description

Technical Field

[0001] The present invention belongs to the field of medical artificial intelligence, and in particular relates to a children's TCM cold and heat syndrome auxiliary identification system based on a report generation model. Background Art

[0002] Effectively distinguishing cold and heat syndromes in children has multiple implications. First, the "Eight-Principle Syndrome Differentiation" theory is the foundation of TCM's syndrome differentiation and treatment, and distinguishing between cold and heat syndromes is a crucial component of this approach. For example, when cold pathogens invade a child's body, they may develop a wind-cold cold, manifesting with cold symptoms such as aversion to cold, loss of sweat, and headaches. Conversely, when heat pathogens invade a child's body, they may develop a wind-heat cold, manifesting with heat symptoms such as fever, sore throat, and thirst. Distinguishing between cold and heat syndromes in children allows for a preliminary assessment of the underlying cause and provides a decisive direction for subsequent treatment. Second, TCM treats illnesses based on syndrome differentiation and treatment, adhering to the principle of "treating heat with cold and treating cold with heat." Failure to accurately distinguish between cold and heat syndromes can lead to inappropriate prescriptions and medications. For example, warming and hot herbs are often used to treat cold-presence syndromes; misusing cooling and cold treatments can exacerbate the condition. Heat-clearing methods are often used to treat heat-presence syndromes; misusing warming and hot treatments can exacerbate the condition. Accurately identifying a child's cold or heat syndrome helps doctors choose the appropriate treatment for optimal results. Finally, children are still in their growth and developmental stages, with their internal organs intact but not yet fully developed. Their resistance to disease is relatively weak, and they are more susceptible to disease transmission. Early identification of a child's cold or heat syndrome helps eliminate pathogenic factors promptly, prevent the disease from spreading, and improve recovery.

[0003] Therefore, the identification of cold and heat syndromes in children in Traditional Chinese Medicine has many meanings, such as clarifying the cause of the disease, guiding treatment and preventing the deterioration of the disease, and has important clinical value.

[0004] To address this challenge, many researchers have adopted multimodal data integration to build models based on clinical experience. For example, Chinese patent publication CN117197836A discloses a method for TCM constitution identification based on deep fusion of multimodal features; and Chinese patent publication CN108986912A discloses a method for intelligently processing tongue images for TCM gastric diseases based on deep learning.

[0005] However, the common TCM multimodal diagnostic model uses an image analysis model to extract high-dimensional image features from visual diagnosis images, and then performs mixed learning with other modal features, resulting in the model having no interpretability on visual diagnosis images. Summary of the Invention

[0006] The present invention provides a children's TCM cold and heat syndrome auxiliary identification system based on a report generation model, which has high interpretability for visual diagnosis images and improves the accuracy of cold and heat syndrome identification.

[0007] A pediatric TCM cold / heat syndrome auxiliary identification system based on a report generation model comprises a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, wherein the computer memory stores a trained report generation model and a multimodal classification model;

[0008] The report generation model includes an encoding part and a decoding part. During the training process, both the encoding part and the decoding part are divided into three steps. In the first step of the encoding part, the ResNet model is used to extract image features of the child's tongue surface image to obtain the image feature X img ; In the second step, the SENet model is used to perform multi-label learning on the children’s tongue surface images to obtain the class activation map feature F img In the third step, the Transformer model is used to extract features from the expert diagnosis report corresponding to the image, obtaining 1024 tokens to form the token embedding layer Y.

[0009] The first step of the decoding part uses a pseudo self-attention layer to transform the image feature X img , class activation map feature F img The two outputs of the decoding module are input together into the cross attention layer for learning, and finally the generated report is obtained through the linear connection layer and the Sotfmax layer.

[0010] During the application process, the computer processor executes the computer program to implement the following steps: input the child's tongue surface image into the trained report generation model, and convert the image feature vector X obtained in the first step of the encoding part into img The data is directly input into the decoding part to generate a report, and the report is normalized into a key-value pattern. The key-valued clinical information, consultation information and the report generated by the report generation model are then input into the multimodal classification model to obtain the classification results of the children's TCM cold and heat syndromes.

[0011] The present invention first analyzes the tongue surface image of children, uses a region-guided report generation model to automatically generate a TCM syndrome differentiation report for the tongue part, and combines the consultation information and clinical information to realize the classification of TCM cold and heat syndrome types of children through a multimodal classification algorithm.

[0012] Furthermore, in the second step of the encoding part, multi-label refers to the classification of tongue diagnosis content, including: tongue color, tongue size, tongue coating thickness and color, tooth marks on the edge of the tongue, and others.

[0013] In the third step of the coding part, the expert diagnosis report is a standardized key-value model. For clinical information and consultation information, keywords are first extracted, and a Chinese medicine pediatric expert sets several states for each keyword. Then all the information is organized into several key-value pairs, and each key-value pair is a state of a keyword.

[0014] The first step of the decoding part is as follows:

[0015]

[0016]

[0017] Where PSA represents the pseudo self-attention layer, Y represents the token embedding, and X img represents the image features, F img Represents the class activation map feature, U k and U v are the newly initialized key and value projection parameters, W q 、W k 、W v Projection parameters for query, key, and value.

[0018] In the second step of the decoding part, the decoding module consists of 24 identical decoding components, each of which includes a self-attention layer and a feedforward propagation layer.

[0019] The self-attention layer is W extracted from the PSA layer. q 、W k 、W v As input, then calculate the v The correlation between the vectors; when extracting the i-th vector v i With the jth vector v j When the correlation between q The query vector q in i Multiply by W k The key vector k in j , the result is the correlation score S between the i-th and j-th vectors (i,j) , the formula is as follows:

[0020] S (i,j) =q i *k j

[0021] After obtaining the correlation score S between the i-th vector and all vectors, the attention distribution weight S' is calculated using softmax (i,j) , thereby realizing the normalization of the relevance score, and through this normalized weight matrix we can see which vectors are related; then, through this attention distribution weight S' (i,j) From Wv Extract important information from the , thereby realizing the self-attention mechanism, the formula is as follows:

[0022]

[0023] Among them, b i That is, according to v i Extracted key information.

[0024] During the training of the report generation model, the cross-entropy loss function is used to continuously reduce this metric through training. The model uses similarity based on word embedding vectors to evaluate the quality of generated reports. This metric calculates the feature vectors of the expert diagnosis report text and the model-generated report, and then uses the cosine value between the feature vectors to evaluate the similarity between the generated report and the expert diagnosis report.

[0025] When the model training achieves the expected results, experts will conduct another evaluation to ensure that the generated report meets the expectations of TCM diagnosis.

[0026] The multimodal classification model adopts the SVM model, which is a model for finding hyperplane decision boundaries. First, the key-valued clinical information, consultation information and the report generated by the child's tongue image of each patient case are regarded as a feature sequence X with several features. i , and then find a weight sequence W of equal length svm And bias constant sequence b to construct a decision hyperplane;

[0027] During training, the goal is to optimize the distance between the decision hyperplane and the characteristic sequences of children with cold and heat syndromes. The formula is as follows:

[0028]

[0029] Among them, X i and y i Represent the characteristic sequence of the i-th patient case and the corresponding cold syndrome and heat syndrome classification labels, W svm and b represent the weight sequence and bias constant sequence respectively.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. This invention innovatively generates a Traditional Chinese Medicine (TCM) inspection report from children's inspection images, and then integrates them with interview information, clinical information, and other multimodal information for learning. Common TCM multimodal diagnostic models typically extract high-dimensional image features from inspection images using image analysis models, and then perform hybrid learning with other modal features, resulting in a lack of interpretability for the inspection images. This invention, however, offers a high degree of interpretability for inspection images, facilitating the subsequent commercialization of the model and its subsequent application and promotion.

[0032] 2. This invention innovatively employs a region-guided report generation model to analyze children's visual diagnosis images. This model can simulate a professional TCM pediatrician, generating specialized TCM visual diagnosis reports for images of a designated area. Converting the images into TCM terminology is more intuitive and clear, effectively improving the model's accuracy in subsequent cold and heat syndrome differentiation. Furthermore, this model can be subsequently applied to other regions, such as the bridge of the nose, cheeks, and lower eyelids, thereby obtaining more comprehensive and rich information from visual diagnosis images and promoting the development of other TCM diagnostic models. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flowchart of an implementation of a system for auxiliary identification of cold and heat syndromes in traditional Chinese medicine for children based on a report generation model according to an embodiment of the present invention;

[0034] Figure 2 A structural diagram of a report generation model according to an embodiment of the present invention;

[0035] Figure 3 2 is a structural diagram of a multimodal classification model in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0037] A children's TCM cold and heat syndrome auxiliary identification system based on a report generation model includes a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor. The computer memory stores a trained report generation model and a multimodal classification model.

[0038] like Figure 1 As shown in the figure, the implementation process of the entire system is as follows:

[0039] 1. Data Collection

[0040] The center collected information on pediatric patients receiving TCM treatment, including clinical information, consultation information, and visual examination images after privacy was removed. The visual examination images were collected using the TFDA-1 tabletop tongue and surface diagnostic instrument from the Shanghai University of Traditional Chinese Medicine Intelligent Diagnostic Technology Research Laboratory.

[0041] 2. Data preprocessing

[0042] For the visual diagnosis images, a professional TCM pediatrician annotated the tongue area with a rectangular box. At the same time, two TCM pediatricians with more than 10 years of clinical experience wrote a report, including the following:

[0043] (1) Tongue color: red, light red or light white, etc.

[0044] (2) Tongue size: symmetrical, fat or thin, etc.

[0045] (3) The thickness and color of the tongue coating: thin and white, thick and white, or greasy yellow, etc.

[0046] (4) Tooth marks on the tongue: no tooth marks, shallow tooth marks, or deep tooth marks, etc.

[0047] (5) Others: peeling, ecchymosis or cracks, etc.

[0048] When the two experts disagreed, a more senior chief physician of Traditional Chinese Medicine Pediatrics would make the diagnosis, and the chief physician's diagnosis would be the final result.

[0049] For clinical and consultation information, keywords were first extracted. Two pediatric TCM experts assigned several states to each keyword. All information was then organized into key-value pairs, with each pair representing a state for each keyword. This approach highly standardized all information, facilitating subsequent model development and report generation.

[0050] 3. Report generation model construction

[0051] The report generation model consists of two parts: encoding and decoding.

[0052] The encoding part is divided into three steps. The first step is to use the ResNet model to extract the image features of the tongue area image and obtain the image feature X img The ResNet model used is a mature model trained on ImageNet. By freezing the weights, a model that can extract image features is obtained.

[0053] In the second step, the SENet model is used to perform multi-label learning on the tongue area image. The multi-label here refers to the tongue diagnosis content classification in data preprocessing: tongue color, tongue size, tongue coating thickness and color, tongue edge tooth marks, and others. After the model is built, the class activation map of the image can be obtained and converted into the class activation map feature F img .

[0054] In the third step, the Transformer model is used to extract features from the report corresponding to the image, obtaining 1024 tokens to form a token embedding layer Y, which is input into the subsequent self-attention layer. The formula is as follows:

[0055] SA(Y)=softmax((YW q )(YW k ) T )(YW v )

[0056] Where SA represents the self-attention layer, Y represents the token embedding, and W q 、W k 、W v Projection parameters for query, key, and value.

[0057] The decoding part is divided into three steps. In the first step, the pseudo self-attention layer is used to transform the image features X img and class activation map features F img They are fused with the token embedding layer Y formed by the report and input into the subsequent self-attention layer. The formula is as follows:

[0058]

[0059]

[0060] Where PSA represents the pseudo self-attention layer, Y represents the token embedding, and X img represents the image features, F img Represents the class activation map feature, U k and U v are the corresponding (newly initialized) key and value projection parameters, W q 、W k 、W v Projection parameters for query, key, and value.

[0061] In the second step, the obtained fusion tokens PSA(X,Y) and PSA(F,Y) are input into the decoding module respectively. The decoding module consists of 24 identical decoding components, each of which includes a self-attention layer and a feedforward propagation layer. The self-attention layer is based on the W extracted by the PSA layer. q 、W k 、W vAs input, then calculate the v The correlation between the vectors. When extracting the i-th vector v i With the jth vector v j When the correlation between q The query vector q in i Multiply by W k The key vector k in j , the result is the correlation score S between the i-th and j-th vectors (i,j) , the formula is as follows:

[0062] S (i,j) =q i *k j

[0063] After obtaining the correlation score S between the i-th vector and all vectors, the attention distribution weight S' can be calculated using softmax (i,j) , thereby realizing the normalization of the correlation score. Through this normalized weight matrix, we can see which vectors have a correlation degree. The normalization formula is as follows:

[0064]

[0065] Then, through this attention distribution weight S' (i,j) From W v Extract important information from the dataset to realize the self-attention mechanism. The formula is as follows:

[0066]

[0067] Among them, b i That is, according to v i Extracted key information.

[0068] In the third step, the outputs of the above two parts are input together into the cross attention layer for learning, and finally the generated report is obtained through the linear connection layer and the Sotfmax layer.

[0069] 4. Report generation model training

[0070] During model training, the visual image, tongue region coordinates, and the corresponding diagnostic report are input simultaneously. The cross-entropy loss function is used as the model's loss function. This metric is continuously reduced during training. The model uses a word embedding-based similarity metric to evaluate the quality of the generated report. This metric is calculated by calculating the feature vectors of the expert's diagnostic report text and the model-generated report, and then using the cosine value between the feature vectors to assess the similarity between the generated report and the expert's diagnostic report.

[0071] When the model training achieves the expected results, experts will conduct another evaluation to ensure that the generated report meets the expectations of TCM diagnosis.

[0072] After training, for visual images that do not provide expert diagnostic reports, reports will be automatically generated. This process is different from the training process. The model will no longer provide the second and third steps of the encoding part. The image feature vector X obtained from the first step is directly input into the decoding part to generate a diagnostic report. The diagnostic report is then normalized into a key-value mode.

[0073] 5. Multimodal classification model construction and training

[0074] The present invention uses the SVM (Support Vector Machine) model to integrate and learn the keyed clinical information, interview information, and reports generated by visual diagnosis images. The SVM model is a model that finds the hyperplane decision boundary. First, the above information of each case is regarded as a feature sequence X with several features. i , and then find a weight sequence W of equal length svm (This is a shared weight) and a bias constant sequence b to construct a decision hyperplane.

[0075] Before training, all data were randomly divided into training set and test set in a ratio of 8:2.

[0076] During training, the goal is to optimize the distance between the decision hyperplane and the characteristic sequences of children with cold syndrome and heat syndrome. The formula is as follows:

[0077]

[0078] Among them, X i and y i Represent the characteristic sequence of the i-th patient case and the corresponding cold syndrome and heat syndrome classification labels, W svm and b represent the weight sequence and bias constant sequence respectively.

[0079] 6. Model Performance Evaluation

[0080] In order to verify the performance of the model in the test set, this paper intends to use precision, recall and F1 indicators for evaluation. The precision formula is as follows:

[0081]

[0082]

[0083]

[0084] Among them, TP represents true positive, that is, the number of a certain class that is accurately predicted to be this class; FP represents false positive, that is, the number of non-a certain class that is incorrectly predicted to be this class; FN represents false negative, that is, the number of a certain class that is incorrectly predicted to be non-this class.

[0085] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A system for assisting in distinguishing cold and heat syndromes in children in Traditional Chinese Medicine based on a report generation model, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that: The computer memory stores a trained report generation model and a multimodal classification model; The report generation model includes an encoding part and a decoding part. During the training process, both the encoding part and the decoding part are divided into three steps. In the first step of the encoding part, the ResNet model is used to extract image features of the child's tongue surface image to obtain the image feature X img ; In the second step, the SENet model is used to perform multi-label learning on the children’s tongue surface images to obtain the class activation map feature F img , where multi-label refers to the classification of tongue diagnosis content, including: tongue color, tongue size, tongue coating thickness and color, tongue teeth, and others; the third step is to use the Transformer model to extract features from the expert diagnosis report corresponding to the image, and obtain 1024 tokens, thereby forming a token embedding layer Y; where the expert diagnosis report is a standardized key-value model. For clinical information and consultation information, keywords are first extracted, and a pediatric expert in traditional Chinese medicine sets several states for each keyword. Then all the information is organized into several key-value pairs, and each key-value pair is a state of a keyword; The first step of the decoding part uses a pseudo self-attention layer to transform the image feature X img , class activation map feature F img They are fused with the token embedding layer Y formed by the report; the second step is to obtain the fusion token PSA (X img ,Y)、PSA(F img ,Y) are input into the decoding module respectively; in the third step, the two parts of the output corresponding to the decoding module are input into the cross attention layer for learning, and finally the generated report is obtained through the linear connection layer and the Sotfmax layer; The first step of the decoding part is as follows: Where PSA represents the pseudo self-attention layer, Y represents the token embedding, and X img represents the image features, F img Represents the class activation map feature, U k and U v are the newly initialized key and value projection parameters, W q 、W k 、W v projection parameters for query, key, and value; During the application process, the computer processor implements the following steps when executing the computer program: inputting the child's tongue surface image into the trained report generation model, directly inputting the image feature vector X obtained in the first step of the encoding part into the decoding part to generate a report, and normalizing the report into a key-value pattern; then inputting the key-valued clinical information, consultation information and the report generated by the report generation model into the multimodal classification model to obtain the classification results of the child's TCM cold and heat syndrome.

2. The pediatric TCM cold and heat syndrome auxiliary identification system based on the report generation model according to claim 1 is characterized in that: In the second step of the decoding part, the decoding module consists of 24 identical decoding components, each of which includes a self-attention layer and a feedforward propagation layer.

3. The pediatric TCM cold and heat syndrome auxiliary identification system based on the report generation model according to claim 2 is characterized in that: The self-attention layer is W extracted from the PSA layer. q 、W k 、W v As input, then calculate the v The correlation between the vectors; when extracting the i-th vector v i With the jth vector v j When the correlation between q The query vector q in i Multiply by W k The key vector k in j , the result is the correlation score S between the i-th and j-th vectors (i,j) , the formula is as follows: S (i,j) =q i *k j After obtaining the correlation score S between the i-th vector and all vectors, the attention distribution weight S′ is calculated using softmax (i,j) , thereby realizing the normalization of the relevance score, and through this normalized weight matrix we can see which vectors are related; then, through this attention distribution weight S′ (i,j) From W v Extract important information from the , thereby realizing the self-attention mechanism, the formula is as follows: Among them, b i That is, according to v i Extracted key information.

4. The pediatric TCM cold and heat syndrome auxiliary identification system based on the report generation model according to claim 1 is characterized in that: During the training of the report generation model, the cross-entropy loss function is used to continuously reduce this metric through training. The model uses similarity based on word embedding vectors to evaluate the quality of generated reports. This metric calculates the feature vectors of the expert diagnosis report text and the model-generated report, and then uses the cosine value between the feature vectors to evaluate the similarity between the generated report and the expert diagnosis report. When the model training achieves the expected results, experts will conduct another evaluation to ensure that the generated report meets the expectations of TCM diagnosis.

5. The pediatric TCM cold and heat syndrome auxiliary identification system based on the report generation model according to claim 1 is characterized in that: The multimodal classification model adopts the SVM model, which is a model for finding hyperplane decision boundaries. First, the key-valued clinical information, consultation information and the report generated by the child's tongue image of each patient case are regarded as a feature sequence X with several features. i , and then find a weight sequence W of equal length svm And bias constant sequence b to construct a decision hyperplane; During training, the goal is to optimize the distance between the decision hyperplane and the characteristic sequences of children with cold and heat syndromes. The formula is as follows: Among them, X i and y i Represent the characteristic sequence of the i-th patient case and the corresponding cold syndrome and heat syndrome classification labels, W svm and b represent the weight sequence and bias constant sequence respectively.

Images