Interrogation method and device for assisting traditional Chinese medicine, equipment and storage medium

Abstract

The invention provides an inquiry method and device for assisting traditional Chinese medicine, equipment and a storage medium. The method comprises the steps of obtaining image information collected by inspection equipment, performing recognition based on the image information, generating an inspection result, obtaining sound signals collected by auscultation information, and classifying the sound signals to generate a sound classification result; pulse condition information collected by a pulse instrument is obtained, feature extraction and analysis are conducted on the pulse condition information, and a pulse condition classification result is generated; obtaining an inquiry text input by the patient on the touch device, and fusing the inquiry text, the inspection result, the sound classification result and the pulse condition classification result to generate a format text; and inputting the format text into the traditional Chinese medicine knowledge graph to generate a text prompt, calling a pre-trained traditional Chinese medicine large language model, inputting the text prompt, and generating an auxiliary diagnosis result. The problem that in the processing and analysis process of traditional Chinese medicine four-diagnosis data, effective fusion and utilization of multi-modal data are lacked, and consequently the accuracy and comprehensiveness of diagnosis results are insufficient is solved.

Description

Technical Field

[0001] The present invention relates to the field of traditional Chinese medical consultation, and in particular to a method, device, equipment and storage medium for assisting traditional Chinese medical consultation. Background Art

[0002] As a traditional medicine, Chinese medicine has a unique theoretical system and diagnosis and treatment methods, and plays an important role in preventing and treating various diseases, especially sub-health and geriatric diseases. However, traditional Chinese medicine diagnosis faces some challenges and difficulties, which are mainly reflected in the following aspects: 1. The diagnostic indicators and standards are not clear, and the diagnostic results are affected by multiple factors of doctors and patients, lacking replicability and consistency. For example, the results of pulse and tongue diagnosis may vary depending on the doctor's level, habits, mood and other factors, and may also vary due to the patient's age, gender, physique, diet, medication and other factors.

[0003] 2. The number and distribution of TCM experts and TCM hospitals are uneven, resulting in an imbalance in the supply and demand of TCM services and uneven quality. For example, it is difficult for TCM's unique techniques and experience to be effectively disseminated and promoted, and some of TCM's advantages and characteristics are difficult to be fully utilized. 3. The diagnosis and treatment process and results of TCM often lack standardized records and preservation, leading to the loss and waste of TCM knowledge and data. For example, classic cases and famous prescriptions of TCM are difficult to fully reproduce and verify, the efficacy and safety of TCM are difficult to be scientifically evaluated and monitored, and the innovation and development of TCM are difficult to be effectively supported and promoted.

[0004] In view of this, this application is filed. Summary of the invention

[0005] The present invention discloses a method, device, equipment and storage medium for assisting traditional Chinese medicine in diagnosis, aiming to solve the problem of lack of effective fusion and utilization of multimodal data in the processing and analysis of traditional Chinese medicine four-diagnosis data, resulting in insufficient accuracy and comprehensiveness of diagnosis results.

[0006] The first embodiment of the present invention provides a method for assisting traditional Chinese medicine in diagnosis, comprising: Acquiring image information collected by the visual diagnosis device, and performing recognition based on the image information to generate a visual diagnosis result, wherein the image information includes a facial image and a tongue image; Acquiring sound signals collected by the olfactory diagnosis information, and classifying the sound signals to generate sound classification results; Acquiring pulse information collected by a pulse monitor, extracting and analyzing features of the pulse information to generate a pulse classification result; Acquire the medical inquiry text input by the patient on the touch device, and fuse the medical inquiry text, the visual examination result, the sound classification result, and the pulse classification result to generate a formatted text; The formatted text is input into the TCM knowledge graph to generate a text prompt, a pre-trained TCM language model is called and the text prompt is input to generate an auxiliary diagnosis result.

[0007] Preferably, the identification is performed based on the image information to generate the visual diagnosis result, specifically: Preprocessing the facial image, wherein the preprocessing includes color correction, image filtering, and image enhancement; Calling a pre-trained facial diagnosis model to extract features from the pre-processed facial image to generate facial features, analyzing and classifying the facial features to generate facial diagnosis results, wherein the facial features include color, gloss, shape, wrinkles, and spots; Preprocessing the tongue image, and performing tongue location and segmentation on the preprocessed tongue image, and retaining the target image; The pre-trained tongue diagnosis model is called to segment and classify the target image, and the tongue diagnosis results are generated based on the tongue image features.

[0008] Preferably, the classifying the sound signal to generate a sound classification result is specifically: Converting the sound signal into a digital signal and preprocessing the digital signal, wherein the sound signal includes breathing, coughing, vomiting, hiccups, belching, sighing, sneezing, and bowel sounds; Sound features are extracted from the digital signal, and a pre-trained olfactory diagnosis model is called to classify the sound features to generate a sound classification result, wherein the sound features include pitch, volume, timbre, and rhythm.

[0009] Preferably, the pulse information is subjected to feature extraction and analysis to generate a pulse classification result, specifically: Converting the pulse information into a digital signal, and pre-processing the digital signal, wherein the pre-processing includes removing noise and baseline drift; The preprocessed digital signal is subjected to feature extraction, and the trained pulse diagnosis model is called to analyze and classify the extraction result to generate a pulse classification result, wherein the extraction result includes time domain features, frequency domain features, morphological features, dynamic features, variability features, and nonlinear features.

[0010] Preferably, the training process of the TCM language model is: Taking LLaMA13B as the basic model, localizing the basic model in Chinese, wherein the localization includes: expanding the vocabulary of LLaMA13B, adding Chinese characters, and performing secondary pre-training using Chinese text data to adapt the model to Chinese; Convert the TCM knowledge graph data into text, merge the converted TCM knowledge graph data and TCM literature data into a text corpus, fine-tune the localized basic model based on the training task and the text corpus, and further fine-tune it through TCM clinical diagnosis and treatment data; Construct a TCM question-and-answer dataset and preprocess the dataset, wherein the question-and-answer dataset is collected, cleaned, annotated, and divided based on the characteristics and needs of TCM; The training is performed by contrasting the loss function. The training process adopts a dynamic learning rate strategy, which is initially 5e-5, and then adjusted according to the progress of training and the performance of the validation set, and finally reduced to 1e-6. The batch value is 64, and the AdamW optimizer is used to automatically adjust the weight attenuation.

[0011] Preferably, it also includes: using the four diagnostic information and labeled data of the corresponding syndrome names for supervised fine-tuning.

[0012] Preferably, it also includes: Randomly extract a portion of data from the instruction fine-tuning dataset as the training set for the reward model. For each piece of training data, use the instruction fine-tuned model to generate several candidate outputs. For each candidate output, a reward value is given, indicating the quality of the output, ranging from 0 to 1. The RLHF algorithm is used to update the model parameters based on the candidate output and the reward value.

[0013] The second embodiment of the present invention provides a medical consultation device for assisting traditional Chinese medicine practitioners, comprising: The inspection result generating unit is used to obtain the image information collected by the inspection device, and perform recognition based on the image information to generate the inspection result, wherein the image information includes a facial image and a tongue image; A sound classification result generating unit, used for acquiring the sound signal collected by the olfactory diagnosis information, and classifying the sound signal to generate a sound classification result; A pulse classification result generating unit is used to obtain the pulse information collected by the pulse monitor, extract and analyze the features of the pulse information to generate a pulse classification result; A format text generating unit is used to obtain the medical inquiry text input by the patient on the touch device, and fuse the medical inquiry text, the visual examination result, the sound classification result, and the pulse classification result to generate a format text; The auxiliary diagnosis result generating unit is used to input the formatted text into the TCM knowledge graph to generate a text prompt, call the pre-trained TCM language model and input the text prompt to generate an auxiliary diagnosis result.

[0014] The third embodiment of the present invention provides a device for assisting Chinese medicine consultation, including a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a method for assisting Chinese medicine consultation as described in any one of the above.

[0015] A fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program can be executed by a processor of a device where the computer-readable storage medium is located to implement a method of assisting traditional Chinese medicine consultation as described in any one of the above items.

[0016] Based on the method, device, equipment and storage medium for assisting TCM consultation provided by the present invention, the image information collected by the inspection device is first obtained, and recognition is performed based on the image information to generate the inspection result, then the sound signal collected by the auscultation information is obtained, and the sound signal is classified to generate the sound classification result; then, the pulse information collected by the pulse imager is obtained, and the pulse information is feature extracted and analyzed to generate the pulse classification result; then, the consultation text input by the patient on the touch device is obtained, and the consultation text, the inspection result, the sound classification result and the pulse classification result are fused to generate a formatted text; finally, the formatted text is input into the TCM knowledge graph to generate a text prompt, the pre-trained TCM language model is called and the text prompt is input to generate the auxiliary diagnosis result. The problem of lack of effective fusion and utilization of multimodal data in the processing and analysis of TCM four-diagnosis data is solved, resulting in insufficient accuracy and comprehensiveness of the diagnosis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a method for assisting traditional Chinese medicine diagnosis provided by the first embodiment of the present invention; Figure 2 This is a schematic diagram of the specific consultation process provided by this paper; Figure 3 It is a module schematic diagram of a device for assisting traditional Chinese medicine diagnosis provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0020] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0022] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0023] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0024] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0025] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0026] The present invention discloses a method, device, equipment and storage medium for assisting traditional Chinese medicine in diagnosis, aiming to solve the problem of lack of effective fusion and utilization of multimodal data in the processing and analysis of traditional Chinese medicine four-diagnosis data, resulting in insufficient accuracy and comprehensiveness of diagnosis results.

[0027] See also Figure 1 and 2 The first embodiment of the present invention provides a method for assisting TCM consultation, which can be performed by a TCM consultation device (hereinafter referred to as the consultation device), and in particular, is performed by one or more processors in the consultation device to implement at least the following steps: S101, acquiring image information collected by an inspection device, and performing recognition based on the image information to generate an inspection result, wherein the image information includes a facial image and a tongue image; In this embodiment, the medical consultation device can be a terminal with data processing capabilities such as a server, workstation, laptop computer, desktop computer, etc., which can communicate with the four diagnostic devices to receive the data collected by them. The medical consultation device can be installed with a corresponding operating system and application software, and the functions required by this embodiment are realized through the combination of the operating system and application software. It should be noted that in other implementation methods, the four diagnostic devices can process the data internally and provide it to the medical consultation device. No specific limitation is made here, but these schemes are all within the scope of protection of the present invention.

[0028] Specifically, in this embodiment, a Chinese medicine intelligent auxiliary diagnosis instrument is used to collect facial images of patients, wherein the image collection can be performed under uniform illumination to ensure that facial details are clearly visible. Among them, facial image preprocessing may include: adjusting the color balance of the image to correct the color distortion caused by uneven illumination to make the facial color more realistic; using image filtering methods such as Gaussian filtering and median filtering to remove noise in the image; performing sharpening and contrast enhancement on the image to enhance the visibility and detail expression of facial features.

[0029] A deep learning model is used to extract features from preprocessed facial images, where the main features include color, gloss, shape, wrinkles, and spots (these features reflect the condition of the body's internal qi and blood, yin and yang, and internal organs). The deep learning model analyzes and classifies the extracted facial features, establishes a facial diagnosis model, and generates detailed text-based facial assessment results based on the combination and changes of facial features, providing a basis for TCM diagnosis.

[0030] Among them, in this embodiment, a data set containing facial images and corresponding TCM facial diagnosis labels is used to train and test the model in the facial diagnosis sub-process. The data source can be the open source face data set CelebA and FLW from research institutions, as well as manually collected facial images. The data format is JPG. A convolutional neural network (CNN) can be used to extract features and classify input facial images. The architecture of the model is the mainstream ResNet facial attribute recognition model. Some indicators, such as accuracy, recall, F1 score, etc., are used to measure the classification effect of the model on different categories.

[0031] Furthermore, in this embodiment, the tongue image is preprocessed (including color correction, image filtering, image enhancement, etc., to improve the clarity and distinguishability of the image), and the tongue body is located and segmented on the preprocessed tongue image, and the target image is retained (i.e., only the tongue portion is retained); Call the pre-trained tongue diagnosis model to segment and classify the target image, and generate tongue diagnosis results based on tongue features (for example, according to tongue categories, such as tongue color, tongue coating, tongue shape, etc., obtain the corresponding TCM diagnosis results, such as yin and yang balance, qi and blood deficiency and excess, five internal organs and six bowels functions, etc., and generate tongue diagnosis results in text form) In this embodiment, a data set containing tongue images and corresponding TCM tongue diagnosis labels can be provided to train and test the model in the tongue diagnosis sub-process. The data source can be an open source tongue image data set from a research institution, as well as manually collected tongue images. Data format: JPG. This embodiment uses deep learning methods, U-Net and residual network (ResNet), to extract tongue image features from images, and classify them according to different tongue image categories, while segmenting the tongue. The model can output multiple tongue image labels, as well as corresponding confidence levels, indicating the model's predicted probability for each label. The model can also handle multi-label situations, that is, a picture may contain multiple tongue image features. Furthermore, in this embodiment, some indicators, such as accuracy, recall, F1 score, etc., are used to measure the classification effect of the model on different categories.

[0032] S102, obtaining sound signals collected by the olfactory diagnosis information, and classifying the sound signals to generate a sound classification result; Specifically, in this embodiment, a microphone or other device is used to collect the user's voice signals, such as breathing, coughing, vomiting, hiccups, belching, sighing, sneezing, bowel sounds, etc. The collected sound signals are converted into digital signals and transmitted to the deep learning model for processing. Filtering technology is used to remove noise from the sound signal and improve the signal quality. The sound signal is resampled to unify the sampling rate of the signal to ensure data consistency. The sound signal is normalized to standardize the signal amplitude to provide better input for subsequent feature extraction and classification.

[0033] Use recurrent neural network (RNN) technologies such as GRU to extract effective features from sound signals, including pitch, volume, timbre, rhythm, etc. Based on the extracted features, different types of sound signals are classified and the types of various sound signals are identified.

[0034] The deep learning method is used to analyze and classify the sound signals and establish a diagnostic model for the sound signals. According to the different types of sound signals, the corresponding TCM diagnosis results are obtained and the sound diagnosis results in text form are generated.

[0035] S103, acquiring pulse information collected by the pulse monitor, extracting and analyzing features of the pulse information to generate a pulse classification result; In this embodiment, the user's pulse wave signal is collected through a sensor on the wrist, including the pulse strength, rhythm, speed and other characteristics. The collected pulse wave signal is converted into a digital signal and transmitted to the deep learning model for processing. Among them, the signal preprocessing may include removing noise from the pulse wave signal and performing baseline drift correction on the pulse wave signal to retain effective features; A recurrent neural network (GRU) is used to extract effective features from the pulse wave signal, and the extracted results are analyzed and classified to generate pulse classification results, wherein the extracted results include time domain features, frequency domain features, morphological features, dynamic features, variability features, and nonlinear features.

[0036] It should be noted that the model can output multiple pulse wave signal labels and corresponding confidence levels, indicating the model's predicted probability for each label. The model can also handle multi-label situations, that is, a pulse wave signal may contain multiple pulse characteristics.

[0037] S104, obtaining a medical inquiry text input by the patient on the touch device, and fusing the medical inquiry text, the visual examination result, the sound classification result, and the pulse classification result to generate a formatted text; It should be noted that the facial diagnosis sub-process, tongue diagnosis sub-process, pulse diagnosis sub-process and questioning sub-process can each diagnose corresponding results from multiple modal human body data. In this embodiment, these multimodal data diagnosis results are innovatively converted into accurate descriptions of formatted text for querying in the TCM knowledge graph and input into the TCM large language model as text prompts. The powerful processing power of the large language model can be used for in-depth analysis and knowledge query. This approach can not only improve the efficiency and accuracy of diagnosis, but also facilitate the teaching and research of TCM.

[0038] Specifically, the content of each diagnostic result obtained from the four examinations is composed of category and confidence level: In tongue diagnosis, the shape of the tongue coating is slender / enlarged; the color is yellow / white; the special characteristics of the tongue, such as cracks, red spots, and thorns; fissured tongue; old tongue; etc.

[0039] During facial diagnosis, there is redness / blackness in the forehead; excessive redness / cyanosis between the eyes; redness / cyanosis at the highest point of the nose bridge; redness / whiteness at the tip of the nose; etc.

[0040] The pitch, tone, timbre, continuity, disease noise, etc. of sounds during auscultation.

[0041] The pulse conditions in pulse diagnosis include floating pulse, sinking pulse, slow pulse, rapid pulse, thin pulse, slippery pulse, and wiry pulse, etc.

[0042] This embodiment will construct a complete and detailed four-diagnosis text prompt at one time based on the results of the four-diagnosis and give a final diagnosis. The steps for setting the prompt are as follows: Organize all the information of the four diagnostic methods of inspection, auscultation, inquiry, and palpation, including all relevant details such as complexion, tongue, pulse, voice, etc. Describe each diagnostic result in detail in the prompt, including the specific phenomena and changes observed. Provide basic information about the patient, such as age, gender, chief complaint, medical history, etc. In the prompt, clearly indicate that you want the model to integrate this information and give a comprehensive TCM diagnosis. Make sure that the description of each diagnostic result is clear and accurate, and avoid expressions that may cause ambiguity. Write different types of diagnostic results in paragraphs so that the model can understand and analyze them more clearly.

[0043] S105, input the formatted text into the TCM knowledge graph to generate a text prompt, call a pre-trained TCM language model and input the text prompt to generate an auxiliary diagnosis result.

[0044] It should be noted that the large language model is a model that uses natural language as input and output. Its knowledge expression and reasoning capabilities mainly rely on the semantics and grammar of the language, rather than logic and mathematics. Without the knowledge graph as a tool for structured and decisive knowledge expression and reasoning, the large language model may find it difficult to handle some complex TCM problems, such as multi-factor etiology analysis, multi-level disease identification, and multi-dimensional treatment plans. Also, without the knowledge graph as a visual and explainable way of knowledge representation and reasoning, the large language model may find it difficult to provide users with a clear and reasonable diagnosis and treatment process and basis; In this embodiment, by combining the TCM knowledge graph and the large language model, the deficiencies of the large language model in TCM knowledge are made up. Furthermore, the knowledge graph is used for structured expression and reasoning of information to ensure that the generated diagnosis results are more accurate and professional.

[0045] In this embodiment, a knowledge graph is constructed to assist the large language model in completing the TCM knowledge diagnosis dialogue. The following knowledge is mainly used: Basic knowledge of TCM: including TCM theory, TCM diagnosis, TCM prescriptions, meridians and acupoints, etc. These knowledge constitute the core part of the TCM knowledge map.

[0046] Disease knowledge: including knowledge on the causes, pathology, symptoms, diagnosis, and treatment of various diseases. This knowledge is very important for assisting large language models in disease diagnosis and recommending treatment plans.

[0047] Symptom description knowledge: Symptoms are manifestations of the patient's condition. Different symptoms may represent different diseases. Therefore, it is necessary to master the description method of symptoms and the relationship between related diseases so that the large language model can accurately understand the patient's condition.

[0048] Drug knowledge: including knowledge about the composition, efficacy, pharmacological effects, side effects, etc. of drugs. This knowledge is very important for assisting the large language model in recommending Chinese medicine prescriptions and providing medication guidance.

[0049] Diagnostic experience knowledge: including the doctor’s own diagnostic experience and case knowledge, which can enhance the diagnostic ability of the large language model and improve its accuracy.

[0050] Literature knowledge: including research results and treatment experience in traditional Chinese medicine literature, which can enrich the knowledge reserve of the large language model and improve its professional level.

[0051] The training process of the vertical domain large language model based on TCM knowledge graph and clinical diagnosis and treatment data is as follows: Base model: LLaMA13B is a large language model trained on a large text corpus. It has 13 billion parameters and can run on a single GPU. LLaMA13B is trained using publicly available datasets without relying on proprietary or inaccessible datasets, ensuring the reproducibility and open source compatibility of its work. LLaMA13B is trained on trillions of tokens and uses multiple data sources from different fields, such as CommonCrawl, C4, and Github; In the first step, the vocabulary of the original LLaMA13B was expanded with Chinese characters to enable it to better encode and decode Chinese text. Then, LLaMA13B was pre-trained twice using 120G of Chinese text data to learn and represent Chinese language knowledge, including data sources in multiple fields such as general, scientific, and literary.

[0052] In the second step, 11 million pieces of TCM knowledge graph data were first converted into text form, for example, (Baihu Tang, treatment method, clearing heat and purging fire) was converted into "Baihu Tang's treatment method is clearing heat and purging fire". Then, the converted knowledge graph data and 1,500 ancient TCM books and literature data were merged into a large text corpus for fine-tuning the base model to learn and represent TCM expertise. Next, different pre-training tasks, such as masked language model, next sentence prediction, multiple choice, etc., were used to train the TCM pre-training model to improve its language understanding and generation capabilities. Finally, 100,000 real TCM expert medical records, 100,000 pulse, tongue, meridian, acupoint data and 2 million real TCM clinical diagnosis and treatment data were used to further fine-tune the TCM pre-training model to adapt it to different TCM application scenarios, such as auxiliary diagnosis and treatment, health conditioning, etc.

[0053] In the supervised fine-tuning stage: First, build a data set. According to the characteristics and needs of traditional Chinese medicine, collect, clean, annotate and divide the medical question-and-answer data set, including questions, answers, knowledge points and other information. The quality and scale of the data set directly affect the effect of the model, so it is necessary to ensure the accuracy, diversity and coverage of the data as much as possible. This stage uses about 100,000 question-and-answer data from traditional Chinese medicine websites, communities, encyclopedias, etc., and about 200,000 question-and-answer data from traditional Chinese medicine literature, encyclopedias, expert consultations, etc. The data set is preprocessed as necessary to remove irrelevant information, segment words, encode, add special symbols, etc. to adapt to the input format of the model.

[0054] Supervised fine-tuning training: The model uses contrast loss as the loss function, which aims to make the answers and knowledge points generated by the model more similar to the correct answers and knowledge points, and more different from the wrong answers and knowledge points. The calculation formula of contrast loss is:

[0055] in, is the batch size, The model generates The answers and knowledge points are the same as The similarity between the correct answer and the knowledge point can be calculated using cosine similarity or other metrics. The advantage of contrast loss is that it can improve the model's ability to distinguish and generalize, and avoid the model from generating repeated or irrelevant answers and knowledge points. Training strategy: The model uses a dynamic learning rate, which is initially 5e-5, and then adjusted according to the progress of training and the performance of the validation set, and finally reduced to 1e-6. The dynamic learning rate can help the model converge quickly in the early stages of training and avoid overfitting and oscillation in the later stages of training.

[0056] The model uses a larger batch size of 64. A larger batch size can increase the stability and efficiency of the model, while also increasing the diversity and robustness of the model because the model can see more data and noise.

[0057] The model uses AdamW as the optimizer, which is an improved Adam optimizer that can automatically adjust the weight decay to reduce the impact of L2 regularization. The advantage of AdamW is that it can adapt to different data and tasks while also maintaining the stability and convergence of the model.

[0058] In the stage of fine-tuning the syndrome name, the TCM syndrome name is a term that summarizes the pathological attributes of the disease development stage in the TCM diagnosis process. It is based on the patient's symptoms, signs, tongue, pulse and other information, combined with TCM theories such as yin and yang, five elements, and organ differentiation.

[0059] In this embodiment, when training the TCM language model, in order to train the model to give the name of the syndrome according to the four diagnoses, labeled data containing detailed four-diagnosis information and corresponding syndrome names were used for supervised fine-tuning. By extracting appropriate four-diagnosis information from historical medical records and electronic medical record diagnoses, combined with the sorting of TCM experts, 10,000 labeled data samples were finally obtained. The following is a training sample of labeled data of four-diagnosis information and corresponding syndrome names that has been de-privatized and simplified: { "Observation": "Pale complexion, slight swelling of eyelids, pale tongue, white tongue coating", "Auscultation": "The voice is deep, there is phlegm, the rhythm is slow, and the breath is weak", "Question": "The patient reported feeling tired and having a loss of appetite", "Pulse diagnosis": "Pulse is deep, thin and weak", "Symptom Name": "Spleen and stomach weakness syndrome" } In this sample, the visual diagnosis information describes the patient's appearance, the auscultation information describes the patient's voice and breathing, the questioning information includes the patient's complaints and feelings, and the palpation information involves the observation of the pulse and tongue. Finally, based on this information, the syndrome name of "weak spleen and stomach syndrome" was derived.

[0060] Finally, this method uses the same training strategy as supervised fine-tuning to fine-tune the TCM language model again.

[0061] The reward model stage refers to the stage of further optimizing the model parameters based on the output generated by the model and the manually labeled reward values ​​using the reinforcement learning method on the basis of fine-tuning the instructions. In the reward model stage, the RLHF algorithm, namely reinforcement learning and hybrid feedforward (RL and Hybrid Feedforward), is used. Combining the policy gradient and supervised learning algorithms, the sparse reward signals are effectively utilized to improve the dialogue generation quality of the model. The specific approach is as follows: First, a portion of data is randomly extracted from the instruction fine-tuning dataset as the training set for the reward model. The data is 1,000 question-and-answer data and 1,000 multi-round dialogue data. Then, for each training data, the model fine-tuned by the instruction is used to generate several candidate outputs. Next, for each candidate output, a reward value is given using manual or automatic methods to indicate the quality of the output, ranging from 0 to 1. Finally, the RLHF algorithm is used to update the model parameters based on the candidate output and the reward value.

[0062] See also Figure 3 The second embodiment of the present invention provides a medical consultation device for assisting traditional Chinese medicine practitioners, comprising: The inspection result generating unit 201 is used to obtain the image information collected by the inspection device, and perform recognition based on the image information to generate the inspection result, wherein the image information includes a facial image and a tongue image; The sound classification result generating unit 202 is used to obtain the sound signal collected by the olfactory diagnosis information, classify the sound signal, and generate a sound classification result; The pulse classification result generating unit 203 is used to obtain the pulse information collected by the pulse monitor, extract and analyze the features of the pulse information to generate a pulse classification result; The format text generating unit 204 is used to obtain the medical inquiry text input by the patient on the touch device, and fuse the medical inquiry text, the visual examination result, the sound classification result, and the pulse classification result to generate a format text; The auxiliary diagnosis result generating unit 205 is used to input the formatted text into the TCM knowledge graph to generate a text prompt, call the pre-trained TCM language model and input the text prompt to generate an auxiliary diagnosis result.

[0063] The third embodiment of the present invention provides a device for assisting Chinese medicine consultation, including a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a method for assisting Chinese medicine consultation as described in any one of the above.

[0064] A fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program can be executed by a processor of a device where the computer-readable storage medium is located to implement a method of assisting traditional Chinese medicine consultation as described in any one of the above items.

[0065] Based on the method, device, equipment and storage medium for assisting TCM consultation provided by the present invention, the image information collected by the inspection device is first obtained, and recognition is performed based on the image information to generate the inspection result, then the sound signal collected by the auscultation information is obtained, and the sound signal is classified to generate the sound classification result; then, the pulse information collected by the pulse imager is obtained, and the pulse information is feature extracted and analyzed to generate the pulse classification result; then, the consultation text input by the patient on the touch device is obtained, and the consultation text, the inspection result, the sound classification result and the pulse classification result are fused to generate a formatted text; finally, the formatted text is input into the TCM knowledge graph to generate a text prompt, the pre-trained TCM language model is called and the text prompt is input to generate the auxiliary diagnosis result. The problem of lack of effective fusion and utilization of multimodal data in the processing and analysis of TCM four-diagnosis data is solved, resulting in insufficient accuracy and comprehensiveness of the diagnosis results.

[0066] Exemplarily, the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the device for implementing a consultation device that assists traditional Chinese medicine. For example, the device described in the second embodiment of the present invention.

[0067] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the method for assisting TCM consultation, and uses various interfaces and lines to connect the various parts of the method for assisting TCM consultation.

[0068] The memory can be used to store the computer program and / or module, and the processor realizes various functions of a method of assisting Chinese medicine consultation by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, a text conversion function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, text message data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0069] Wherein, if the implemented module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0070] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0071] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method of assisting traditional Chinese medicine in diagnosis. It is characterized in that include: Acquiring image information collected by the visual diagnosis device, and performing recognition based on the image information to generate a visual diagnosis result, wherein the image information includes a facial image and a tongue image; Acquiring sound signals collected by the olfactory diagnosis information, and classifying the sound signals to generate sound classification results; Acquiring pulse information collected by a pulse monitor, extracting and analyzing features of the pulse information to generate a pulse classification result; Acquire the medical inquiry text input by the patient on the touch device, and fuse the medical inquiry text, the visual examination result, the sound classification result, and the pulse classification result to generate a formatted text; The formatted text is input into the TCM knowledge graph to generate a text prompt, a pre-trained TCM language model is called and the text prompt is input to generate an auxiliary diagnosis result.

2. A method for assisting traditional Chinese medicine in diagnosis according to claim 1, It is characterized in that The identification is performed based on the image information to generate the visual diagnosis result, specifically: Preprocessing the facial image, wherein the preprocessing includes color correction, image filtering, and image enhancement; Calling a pre-trained facial diagnosis model to extract features from the pre-processed facial image to generate facial features, analyzing and classifying the facial features to generate facial diagnosis results, wherein the facial features include color, gloss, shape, wrinkles, and spots; Preprocessing the tongue image, and performing tongue location and segmentation on the preprocessed tongue image, and retaining the target image; The pre-trained tongue diagnosis model is called to segment and classify the target image, and the tongue diagnosis results are generated based on the tongue image features.

3. A method of assisting traditional Chinese medicine in diagnosis according to claim 1, It is characterized in that The classifying the sound signal to generate a sound classification result is specifically: Converting the sound signal into a digital signal and preprocessing the digital signal, wherein the sound signal includes breathing, coughing, vomiting, hiccups, belching, sighing, sneezing, and bowel sounds; Sound features are extracted from the digital signal, and a pre-trained olfactory diagnosis model is called to classify the sound features to generate a sound classification result, wherein the sound features include pitch, volume, timbre, and rhythm.

4. A method for assisting traditional Chinese medicine in diagnosis according to claim 1, It is characterized in that The pulse information is subjected to feature extraction and analysis to generate a pulse classification result, specifically: Converting the pulse information into a digital signal, and pre-processing the digital signal, wherein the pre-processing includes removing noise and baseline drift; The preprocessed digital signal is subjected to feature extraction, and the trained pulse diagnosis model is called to analyze and classify the extraction results to generate pulse classification results, wherein the extraction results include time domain features, frequency domain features, morphological features, dynamic features, variability features, and nonlinear features.

5. A method for assisting traditional Chinese medicine in diagnosis according to claim 1, It is characterized in that The training process of the TCM language model is as follows: Taking LLaMA13B as the basic model, localizing the basic model in Chinese, wherein the localization includes: expanding the vocabulary of LLaMA13B, adding Chinese characters, and performing secondary pre-training using Chinese text data to adapt the model to Chinese; Convert the TCM knowledge graph data into text, merge the converted TCM knowledge graph data and TCM literature data into a text corpus, fine-tune the localized basic model based on the training task and the text corpus, and further fine-tune it through TCM clinical diagnosis and treatment data; Construct a TCM question-and-answer dataset and preprocess the dataset, wherein the question-and-answer dataset is collected, cleaned, annotated, and divided based on the characteristics and needs of TCM; The training is performed by contrasting the loss function. The training process adopts a dynamic learning rate strategy, which is initially 5e-5, and then adjusted according to the progress of training and the performance of the validation set, and finally reduced to 1e-6. The batch value is 64, and the AdamW optimizer is used to automatically adjust the weight attenuation.

6. A method of assisting traditional Chinese medicine in diagnosis according to claim 5, It is characterized in that Also includes: Use the four diagnostic information and labeled data of the corresponding syndrome names for supervised fine-tuning.

7. A method of assisting traditional Chinese medicine in diagnosis according to claim 6, It is characterized in that Also includes: Randomly extract a portion of data from the instruction fine-tuning dataset as the training set for the reward model. For each piece of training data, use the instruction fine-tuned model to generate several candidate outputs. For each candidate output, a reward value is given, indicating the quality of the output, ranging from 0 to 1. The RLHF algorithm is used to update the model parameters based on the candidate output and the reward value.

8. A device to assist traditional Chinese medicine in diagnosis. It is characterized in that include: The inspection result generating unit is used to obtain the image information collected by the inspection device, and perform recognition based on the image information to generate the inspection result, wherein the image information includes a facial image and a tongue image; A sound classification result generating unit, used for acquiring the sound signal collected by the olfactory diagnosis information, and classifying the sound signal to generate a sound classification result; A pulse classification result generating unit is used to obtain the pulse information collected by the pulse monitor, extract and analyze the features of the pulse information to generate a pulse classification result; A format text generating unit is used to obtain the medical inquiry text input by the patient on the touch device, and fuse the medical inquiry text, the visual examination result, the sound classification result, and the pulse classification result to generate a format text; The auxiliary diagnosis result generating unit is used to input the formatted text into the TCM knowledge graph to generate a text prompt, call the pre-trained TCM language model and input the text prompt to generate an auxiliary diagnosis result.

9. A device to assist traditional Chinese medicine consultation. It is characterized in that It comprises a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a method for assisting traditional Chinese medicine consultation as described in any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that A computer program is stored, and the computer program can be executed by a processor of the device where the computer-readable storage medium is located to implement a method of assisting traditional Chinese medicine consultation as described in any one of claims 1 to 7.

Images