An intelligent traditional Chinese medicine diagnosis system based on an AI large model

Abstract

The application provides an intelligent traditional Chinese medicine diagnosis system based on an AI large model, and relates to the technical field of traditional Chinese medicine auxiliary diagnosis and treatment. The system comprises an information collection module, a data processing module, a diagnosis generation module and a traditional Chinese medicine specific large model diagnosis module. The information collection module collects user information through face recognition, tongue recording, patient self-reports and traditional Chinese medicine palpation. The data processing module is used for preprocessing the collected five kinds of information. The diagnosis generation module diagnoses the data output by the data processing module. The traditional Chinese medicine specific large model diagnosis module takes the comprehensive diagnosis generated by the diagnosis generation module as input, and outputs corresponding scheme suggestions through a pre-trained traditional Chinese medicine specific large model. The five kinds of information collected by the application can be collected online, without the need to go to a designated place or use related machines. On the premise of ensuring the realization of functions, the entire system truly realizes remote diagnosis, and the generation ability of the large model solves the single diagnosis and suggestion problem of the past system.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of auxiliary diagnosis and treatment of traditional Chinese medicine, and in particular to an intelligent traditional Chinese medicine diagnosis system based on an AI large model. BACKGROUND

[0002] Traditional Chinese medicine doctors have a general understanding of the physiological conditions such as the rise and fall of qi and blood, cold and heat, and deficiency and excess of patients through the information of four diagnoses of "observation, listening, interrogation and palpation", summarize the disease conditions, and then use various treatment methods such as traditional Chinese medicine, acupuncture, massage, cupping therapy and diet therapy to cure the patients as a whole.

[0003] With the development of information technology, traditional Chinese medicine diagnosis is gradually combined with information technology. The remote medical diagnosis mode no longer requires offline communication between doctors and patients, greatly facilitating traditional Chinese medicine diagnosis; the traditional Chinese medicine diagnosis data analysis system based on artificial intelligence sends suspected diseases to doctors for review through a diagnosis mapping model; the intelligent traditional Chinese medicine diagnosis system based on deep learning combines the information of four diagnoses, uses a convolutional neural network deep learning technology to train a model for pre-training, and matches the model with a local knowledge base and cases to accurately locate the disease condition of a patient. However, the current information technology combined with traditional Chinese medicine diagnosis has limitations, the number of local traditional Chinese medicine knowledge bases is limited, which may not cover all patient conditions, and the generated disease diagnosis and corresponding treatment plan are generated by matching existing cases, the result is relatively single and lacks pertinence.

[0004] The current AI large model technology uses the decoder of the transformer to generate targeted language descriptions, and through model fine-tuning, the local knowledge base is used as the data set for model fine-tuning to generate a corresponding special large model. The learned knowledge can be interconnected to improve the accuracy of diagnosis and also customize the output of diagnosis, which exactly makes up for the defects of the current traditional Chinese medicine diagnosis system combined with information technology.

[0005] Therefore, it is necessary to develop an intelligent traditional Chinese medicine diagnosis system based on an AI large model to make up for the deficiencies of the current diagnosis system. SUMMARY

[0006] The technical problem to be solved by the application is to provide an intelligent traditional Chinese medicine diagnosis system based on an AI large model to solve the deficiencies of the prior art.

[0007] To solve the above technical problems, the technical scheme adopted by the application is as follows:

[0008] An intelligent traditional Chinese medicine diagnosis system based on an AI large model, comprising an information collection module, a data processing module, a diagnosis generation module and a traditional Chinese medicine special large model diagnosis module.

[0009] The information collection module collects information of the user through four parts of facial recognition, tongue image recording, patient self-reporting and TCM palpation. The facial recognition part and the tongue image recording part obtain facial information and tongue image information through online video recording. The patient self-reporting part obtains relevant conditions of the patient through online video recording, extracts the voice of the video to obtain voice information, and then obtains text information through voice transcription technology. The TCM palpation part obtains a pulse diagnosis heart rate graph through online video recording and a non-contact heart rate measurement system.

[0010] The data processing module is configured to preprocess the five kinds of information collected by the information collection module. For the facial information and the tongue image information, the data processing module extracts one frame of picture per second from the video. For the text information, the data processing module performs word segmentation on the text and converts the text into a numerical form. For the pulse diagnosis heart rate graph, the data processing module converts the pulse diagnosis heart rate graph through FFT to obtain a frequency spectrum graph. For the voice information, the data processing module obtains corresponding voice features through WaveNet.

[0011] The generation diagnosis module diagnoses the data output by the data processing module. The several frames of pictures obtained from the facial information and the tongue image information are respectively input into the corresponding pre-trained VisualGLM model, the tongue image or the facial image is encoded by using a visual encoder, image features are generated, and then the image features are combined with text prompts to generate diagnosis results of the face and the tongue through a self-recurrent decoder. The preprocessed text is input into a pre-trained BERT-over-BERT (BoB) model as a text segment for understanding the role of the user, and then the BoB model is asked questions according to pre-set questions to generate the medical history, symptoms, living environment, sweating condition, diet condition, sleep condition, excretion condition, and whether there is headache, chest pain, waist and abdominal pain, and output a text diagnosis. The frequency spectrum graph generated from the pulse diagnosis heart rate graph is input into a hybrid model CRNN based on CNN and LSTM to calculate the possibility of having related cardiovascular diseases by using a hybrid model ARNN based on time-frequency component attention for heart sound classification, and output a corresponding pulse diagnosis. The voice features generated from the voice information are input into a pre-trained MViTv2 model to output a preliminary classification of the user's voice, including the strength, height, clearness, length, and speed of the user's voice, and then the preliminary classification results are input into a pre-trained SVM model to output a judgment of the patient's lung qi condition, emotional changes, whether the qi and blood are sufficient, whether the spleen and stomach are diseased, and the health degree of the viscera, as a voice diagnosis. Finally, the five kinds of diagnosis are input into a ChatGLM4 model to output a summary text to form a comprehensive diagnosis.

[0012] The TCM special large model diagnosis module takes the comprehensive diagnosis generated by the generation diagnosis module as input, and then outputs a corresponding scheme suggestion through a pre-trained TCM special large model.

[0013] Further, the non-contact heart rate measurement system in the information collection module is composed of information acquisition hardware and data processing software; the process of obtaining the heart rate graph by the non-contact heart rate measurement system is divided into pulse data acquisition part, pulse data processing part, heart rate signal processing part and real-time heart rate and stable heart rate calculation part, and the specific steps are as follows:

[0014] Step 1.1: pulse data acquisition and processing; the non-contact heart rate measurement system shoots the wrist part of the user, extracts features and calculates the ROI (region of interest) green channel average value of the "cun, guan and chi" parts;

[0015] Step 1.2: heart rate signal processing; calculate the frames per second (FPS) of each time the green channel average value is extracted, convert the green channel average value into a matrix and interpolate it to a uniform time interval, and obtain the time division signal of the heart rate information after signal processing; use Fourier transform to convert the time division signal into a frequency division signal;

[0016] Step 1.3: real-time heart rate and stable heart rate calculation; calculate the frequency array by FPS and obtain the heart rate spectrum graph by the amplitude spectrum of the frequency division signal; calculate the real-time heart rate by the peak value of the spectrum graph in a reasonable range, and then calculate the stable heart rate.

[0017] Further, in step 1.1, the specific method of the pulse data acquisition part is: first, the user selects to upload a video or shoot a video in real time according to the guidance, correctly places the non-contact heart rate measurement system, turns on the built-in illumination lamp of the non-contact heart rate measurement system, and ensures that the light source is stable; then, the user is required to place the wrist in the designated area of the non-contact heart rate measurement system in the video, maintain a certain posture stability to ensure that the key features around the pulse are recorded completely; after obtaining qualified video data, the non-contact heart rate measurement system will automatically capture a specified number of key frames from the video stream based on the clarity and integrity of the wrist feature display;

[0018] In step 1.1, the specific method of the pulse data processing part is: the system first identifies the ROI of the wrist, which is described by the following formula: (x, y) → ROI, where (x, y) represents the coordinates of the center of the ROI; the system marks the wrist area in the image and divides it, ensuring that only this area is analyzed and cropped in the subsequent steps, and the specific steps are as follows:

[0019] S1.1.1: identify the boundary coordinates of the wrist area and divide it into three parts: cun, guan and chi;

[0020] S1.1.2: Color space conversion and mask processing are performed on each part respectively to ensure that only the skin area is retained;

[0021] S1.1.3: The maximum contour of the skin area is found, and ROI extraction is performed based on this;

[0022] S1.1.4: For images in which the wrist area is successfully identified, the average value of the green channel is further extracted as the basis data of the heart rate signal.

[0023] Further, the specific method of step 1.2 is as follows:

[0024] S1.2.1: Color extraction and standardization; the average value of the green channel in the RGB color channel is extracted from the ROI after image processing, and standardization processing is performed;

[0025] The extracted average value is:

[0026]

[0027] Where G is the average value of the green channel, g i is the green value of the ith pixel, and N is the total number of pixels;

[0028] The green channel average value is normalized as follows:

[0029]

[0030] Where Normalized_data is the normalized green channel average value, and G is the original green channel average value;

[0031] S1.2.2: Signal detrending and interpolation; after obtaining the green channel average value of the pulse ROI, the detected values are stored in the data buffer in the form of a queue, and when the buffer is full, the data in the buffer is converted to a matrix signal. Then, the signal data is detrended. The timestamp of each detection of the green channel average value is calculated to obtain the actual FPS, i.e., the number of detections per second. According to the periodic time sequence obtained by FPS, the green channel average value is interpolated into a sequence with an average time interval to obtain the time division signal of the heart rate information;

[0032] S1.2.3: Fast Fourier transform; the time division signal is subjected to fast Fourier transform to obtain a frequency division signal. Then, a bandpass filter is used for frequency spectrum analysis based on the standard range of heart rate: 50 to 180, to screen out the heart rate signal within a specific frequency range. The Fourier transform formula is as follows:

[0033] FFT = |F(N t *30)| 2

[0034] where N t is a normalized time domain signal, F is a Fourier transform operator:

[0035]

[0036] where x(t) is a time domain signal, t is time, X(f) is a frequency domain signal, and f is a signal frequency.

[0037] Further, the specific method of step 1.3 is:

[0038] According to the transformed frequency domain signal, the frequency array is calculated as follows:

[0039]

[0040] where freqs_bpm[k] is the result of converting the frequency array to beats per minute, k is the frequency index, fps is the sampling frequency, and D is the length of the signal.

[0041] Performing spectral analysis on the frequency array, filtering out heart rates within the standard range, and obtaining a heart rate spectrum; finding the peak value of the frequency in the spectrum, which is the current real-time heart rate; averaging all real-time heart rates measured historically, and calculating the stable heart rate value within a period of time as follows:

[0042]

[0043] where bpms represents the stable heart rate, K is the number of historical heart rate data, and bpm i is the i-th calculated heart rate value.

[0044] The final system provides the user with the corresponding heart rate graph, real-time heart rate, and stable heart rate in the "cunkuanchi" region.

[0045] Further, the generation of facial diagnosis and tongue diagnosis in the generation diagnosis module is divided into data collection, data processing, VisualGLM fine-tuning, and VisualGLM diagnosis, and the specific steps are as follows:

[0046] Step 2.1: Data collection; the AI-based intelligent traditional Chinese medicine diagnosis system guides the user to collect facial or tongue data through online video in the form of facing the camera, opening the mouth, and sticking out the tongue.

[0047] Step 2.2: Data processing; using known YOLOX and Segment Anything-based open-source TongueSAM model, generate boundary box prompts through image encoder, prompt generator and mask decoder to achieve high-precision tongue image segmentation, and extract the face and tongue in the collected video data respectively;

[0048] Step 2.3: VisualGLM fine-tuning; the system uses LoRA (Low-Rank Adaptation) method to fine-tune the multi-modal dialogue language model VisualGLM model;

[0049] Step 2.4: Processed face images or tongue images are sent into the VisualGLM model for processing in combination with specific prompts; VisualGLM-6B encodes the input image through the image encoder, and generates the description and analysis results of the corresponding tongue image or face image in combination with the prompts generated by the prompt generator.

[0050] Further, in step 2.2, for the face diagnosis module, the system uses YOLOX to extract the face part in the image as the region of interest, i.e. ROI; (x, y) represents the coordinates of the center of ROI, w and h represent the width and height of ROI respectively; then the boundary coordinates of the ROI region are determined, (x1, y1) represents the left upper corner coordinates, and (x2, y2) represents the right lower corner coordinates, then

[0051]

[0052]

[0053] For the tongue diagnosis module, first use YOLOX to label the ROI containing the tongue, and then for the images successfully recognized as tongue regions, use TongueSAM algorithm to calculate the probability of each pixel point belonging to a class, and extract the tongue part in the image.

[0054] Further, the specific method of step 2.3 is:

[0055] The system first prepares a set of training data D = {(x i , t i )} composed of face images and corresponding text descriptions or tongue images and corresponding text descriptions, where x i represents a face image or a tongue image, and t i represents the corresponding text description.

[0056] Use the visual encoder E v of the VisualGLM model to process the image data to obtain the feature representation v of the imagei = E v (x i );text encoder E using VisualGLM model t process the textual description to obtain text feature t i ′ = E t (t i );

[0057] Introduce a low-rank matrix W to update the weights of the model, and combine the image feature v i and the text feature t i ′ with the low-rank matrix W to calculate the model output y i :

[0058] y i = f(v i ,t i ′ ; θ, W)

[0059] where f represents the forward propagation function of the model, and y i is the prediction result output by the model according to the input v i and t i ′ ;

[0060] Through supervised learning based on minimizing the loss function L between the prediction result y i and the real textual description t i , the difference between the prediction result and the real value is minimized:

[0061]

[0062] where I is the loss function for measuring the difference between the prediction result and the real textual description.

[0063] Further, the TCM special large model diagnosis module sets a prompt word through a pre-trained model, analyzes the possible disease of the user according to the information of comprehensive diagnosis, judges the constitution of the user according to the characteristics of the eight constitutions of TCM, outputs the place where the user coincides with the typical characteristics in the eight constitutions of TCM and the characteristics of the corresponding constitution, analyzes the possible disease risk of the current user according to the above comprehensive information, and gives corresponding suggestions according to the constitution of the user and the possible disease;

[0064] The pre-trained model is a large model with TCM special knowledge obtained by training a large number of diagnosis cases in the early stage and incorporating the judgment of the eight constitutions of TCM.

[0065] The beneficial effects produced by the technical scheme are that the intelligent traditional Chinese medicine diagnosis system based on an AI large model provided by the application can collect all the five kinds of information online without going to a designated place or using related machines, realizes remote diagnosis of the whole system under the premise of realizing functions, uses a pre-trained VisualGLM model to process facial information and tongue information, inputs pictures, and outputs more targeted and rich diagnoses, judges the constitution of a user through a pre-trained traditional Chinese medicine special large model, analyzes the possible diseases of the user, and gives corresponding suggestions. The application solves the problem of single diagnosis and suggestion of the past system by using the generation ability of the large model. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 A system structure block diagram of the intelligent traditional Chinese medicine diagnosis system based on an AI large model provided by the embodiment of the application is provided.

[0067] Figure 2 A process block diagram of pulse diagnosis using a heart rate detection system provided by the embodiment of the application is provided.

[0068] Figure 3 A process block diagram of facial diagnosis or tongue image diagnosis using VisualGLM provided by the embodiment of the application is provided. DETAILED DESCRIPTION

[0069] The specific embodiments of the application are described in further detail below in combination with the drawings and examples. The following examples are used to illustrate the application, but are not used to limit the scope of the application.

[0070] As shown in the drawings, Figure 1 The intelligent traditional Chinese medicine diagnosis system of the embodiment is divided into four modules: an information collection module, a data processing module, a diagnosis generation module, and a traditional Chinese medicine special large model diagnosis module.

[0071] The information collection module collects the information of a user through four parts: facial recognition, tongue image recording, patient self-reporting, and traditional Chinese medicine palpation. The facial recognition part obtains facial information by online video recording, the tongue image recording part obtains tongue image information by online video recording, the patient self-reporting part obtains the related diseases of a patient by online video recording, extracts the voice of the video to obtain voice information, and then obtains text information through voice transcription technology, and the traditional Chinese medicine palpation part obtains a pulse diagnosis heart rate graph by using a non-contact heart rate measurement system through online video.

[0072] As shown in the drawings, Figure 2As shown, the pulse diagnosis heart rate graph obtained by the non-contact heart rate measurement system composed of information collection hardware and data processing software is divided into pulse data collection part, pulse data processing part, heart rate signal processing part and real-time heart rate and stable heart rate calculation part, and the specific steps are as follows:

[0073] Step 1.1: Pulse data collection and processing; the non-contact heart rate measurement system extracts features from the "cun, guan, chi" parts and calculates the average value of the ROI green channel by shooting the user's wrist part. The specific method is as follows:

[0074] Pulse data collection part: first, the user selects to upload the video or real-time video shooting according to the guidance, correctly places the measuring instrument, and turns on the illumination lamp to ensure stable light source. Then, the user is required to place the wrist in the designated area of the instrument in the video, maintain a certain posture stability to ensure that the key features around the pulse are recorded completely. After obtaining qualified video data, the system will automatically capture a specified number of key frames from the video stream based on the clarity and integrity of the wrist feature display.

[0075] Pulse data processing part: the system first identifies the region of interest of the wrist. This process can be described by the following formula: (x, y)→ROI, where (x, y) represents the coordinates of the center of ROI. The system marks and divides the wrist area in the image to ensure that only this area is analyzed and cropped in the subsequent steps. The specific steps are as follows:

[0076] S1.1.1: Identify the boundary coordinates of the wrist area and divide it into three parts: cun, guan, chi;

[0077] S1.1.2: Perform color space conversion and mask processing on each part respectively to ensure that only the skin area is retained;

[0078] S1.1.3: Find the largest contour of the skin area and extract ROI based on it;

[0079] S1.1.4: For images where the wrist area is successfully identified, the system will further extract the average value of the green channel as the basis data of the heart rate signal.

[0080] Step 1.2: Heart rate signal processing; calculate the number of frames per second (FPS) for each extraction of the green channel average value, matrix the green channel average value and interpolate to a uniform time interval, and obtain the time signal of the heart rate information after signal processing; use Fourier transform to convert the time signal to frequency signal. The specific method is as follows:

[0081] S1.2.1, color extraction and standardization. The average value of the green channel in the RGB color channel is extracted from the ROI after image processing, and standardized.

[0082] Extract the average value:

[0083]

[0084] Where G is the average value of the green channel, g i is the green value of the ith pixel, and N is the total number of pixels.

[0085] The average value of the green channel is normalized:

[0086]

[0087] Where Normalized_data is the normalized average value of the green channel, and G is the original average value of the green channel.

[0088] S1.2.2, signal detrending and interpolation. After obtaining the average value of the green channel of the pulse ROI, the detected values are stored in the data buffer in the form of a queue. When the buffer is full, the data in the buffer is converted to a matrix signal. Then the signal data is detrended to eliminate the overall trend. Calculate the timestamp of each detection of the average value of the green channel to get the actual FPS, i.e. the number of detections per second. Here is not the FPS of the camera, but the actual detection FPS. According to the periodic time series obtained by FPS, the average value of the green channel is interpolated into a sequence of average time intervals, thus obtaining the time division signal of heart rate information.

[0089] S1.2.3, fast Fourier transform. The time division signal is subjected to fast Fourier transform to obtain the frequency division signal. Then, according to the standard range of heart rate: 50 to 180, a band-pass filter is used for frequency spectrum analysis to screen out the heart rate signal in a specific frequency range. The Fourier transform formula is as follows:

[0090] FFT = |F(N t *30)| 2

[0091] Where N t is the normalized time domain signal, and F is the Fourier transform operator:

[0092]

[0093] Where x(t) is the time division signal, t is the time, X(f) is the frequency domain signal, and f is the signal frequency.

[0094] Step 1.3: Real-time heart rate and stable heart rate calculation part: Calculate the frequency array by FPS, and combine the amplitude spectrum of the frequency-division signal to get the heart rate spectrum map; calculate the real-time heart rate by the peak value of the spectrum map in a reasonable range, and then calculate the stable heart rate. The specific method is as follows:

[0095] According to the transformed frequency-division signal, the frequency array is calculated:

[0096]

[0097] Wherein, freqs_bpm[k] is the result of converting the frequency array to beats per minute (BPM), k is the frequency index, fps is the sampling frequency, and D is the length of the signal.

[0098] The frequency array is subjected to spectrum analysis, and the heart rate in the standard range (50-180) is selected to obtain the heart rate spectrum map.

[0099] Then find the peak value of the frequency in the spectrum map, which is the current real-time heart rate. Then average all real-time heart rates measured according to history to calculate the stable heart rate value in a period of time:

[0100]

[0101] Wherein, bpms represents the stable heart rate, K is the number of historical heart rate data, and bpm i is the heart rate value calculated for the i-th time.

[0102] Finally, the system provides the corresponding heart rate map, real-time heart rate and stable heart rate in the "cunkuanchi" region to the user.

[0103] The data processing module is responsible for pre-processing the five kinds of information collected. For facial information and tongue information, this module extracts one frame of picture per second from the video; for text information, this module performs word segmentation on the text and converts the text into numerical form; for pulse diagnosis heart rate chart, this module converts by FFT to get the spectrum map; for voice information, this module gets the corresponding voice features through WaveNet (WaveNet).

[0104] The diagnostic module performs diagnostics on the data output from the data processing module. Several frames of images obtained from facial and tongue images are processed by corresponding pre-trained VisualGLM models to output corresponding facial and tongue diagnoses. Initially, a BERT-over-BERT (BoB) model is trained using dialogue between a doctor and patient as training data. Then, the user's text information is input into this pre-trained model to output a text diagnosis. The spectrogram generated from the pulse and heart rate map is input into a pre-trained hybrid model CRNN based on CNN and LSTM. A heart sound classification method based on a hybrid model ARNN using time-frequency component attention is used to calculate the probability of related cardiovascular diseases and output the corresponding pulse diagnosis. Speech features generated by WaveNet are classified using the MViTv2 and SVM models to output a speech diagnosis. Finally, the five diagnoses are input into the ChatGLM4 model to output a summary text, forming a comprehensive diagnosis.

[0105] like Figure 3 As shown, the steps for generating facial and tongue diagnoses using the diagnostic generation module are as follows:

[0106] Step 2.1: Data Acquisition. The system first provides detailed instructions through the user interface, guiding users on how to start an online video session on their own devices. It also ensures good lighting conditions, camera positioning, and a clean background.

[0107] For the facial diagnosis module, users need to keep their face directly facing the camera and maintain a stable posture to ensure that key facial features are fully recorded. For the tongue diagnosis module, users are required to open their mouths and stick out their tongues in the video, maintaining a certain level of postural stability to ensure that key features around the tongue are fully recorded.

[0108] After obtaining qualified video data, the system will automatically capture a specified number of keyframes from the video stream based on the clarity and integrity of facial or tongue features.

[0109] Step 2.2: Data Processing. Using YOLOX and the TongueSAM model based on Segment Anything, the face and tongue are extracted from the acquired video data, respectively.

[0110] For the facial diagnosis module, the system uses YOLOX to extract the face portion of the image as the Region of Interest (ROI). Let (x, y) represent the coordinates of the ROI center, and w and h represent the width and height of the ROI, respectively. Then, the boundary coordinates of the ROI region are determined, with (x1, y1) representing the top-left corner and (x2, y2) representing the bottom-right corner.

[0111]

[0112] Similarly, for the tongue diagnosis module, the system first uses YOLOX to label the ROI containing the tongue. Then, for images where the tongue region is successfully identified, the system uses the open-source TongueSAM algorithm, which uses a pre-trained image encoder to encode the input image, generates bounding box cues based on object detection, and uses a mask decoder to segment and extract the image, achieving high-precision tongue segmentation.

[0113] Step 2.3: VisualGLM Fine-tuning. The VisualGLM model is fine-tuned using the LoRA (Low-Rank Adaptation) method. VisualGLM-6B is a known multimodal dialogue language model that encodes input tongue or facial images using a visual encoder, combines this with prompts generated by a prompt generator, and performs multimodal processing and analysis. Specifically:

[0114] The system first prepares a training dataset D = {(x i ,t i )}, where x i Represents a facial image or a tongue image, t i This represents the corresponding text description.

[0115] In the feature extraction stage, the system uses the visual encoder E of the VisualGLM model. v To process image data and obtain the image's feature representation v i =E v (x i Meanwhile, text encoder E t Used to process text descriptions and obtain text features t i ′ =E t (t i ).

[0116] During the LoRA fine-tuning phase, the system introduces a low-rank matrix W to update the model's weights, incorporating image features v. i and text features t i ′ Combined with the low-rank matrix W, the model output y is calculated. i :

[0117] y i =f(v i ,t i ′ ;θ,W)

[0118] where f represents the forward propagation function of the model, y i is the prediction result output by the model according to the input v i and t i ′ .

[0119] The supervised learning-based training is performed by minimizing the loss function L between the prediction result y i and the real text description t i to minimize the error between the prediction value and the real value:

[0120]

[0121] where I is the loss function, which measures the difference between the prediction result and the real text description.

[0122] Step 2.4: VisualGLM diagnosis. The processed facial image or tongue image is sent into the VisualGLM model along with specific prompts, and the VisualGLM model outputs the facial or tongue image diagnosis result based on the input image and prompts.

[0123] The system analyzes the pre-processed facial or tongue image through the pre-fine-tuned VisualGLM model and generates specific diagnosis results. First, the system inputs the pre-processed facial or tongue image into the VisualGLM model, processes the image data through its visual encoder part, and extracts the visual features of the image. The extracted image features F are then sent to the decoder D of the model. The model combines specific prompts to guide the generation of more targeted diagnosis descriptions. This process is represented by the following formula:

[0124] Y = D(F, P)

[0125] where Y represents the generated diagnosis description, and P is the specific prompt. Finally, the system integrates and formats the generated diagnosis result Y into a diagnosis report.

[0126] The TCM special large model diagnosis module takes the comprehensive diagnosis generated by the diagnosis module as input, and then outputs the corresponding scheme suggestion through the pre-trained TCM special large model. In the early stage, the TCM special large model is trained to have the special knowledge of TCM by a large number of diagnosis cases and the judgment of the eight TCM constitutions. When outputting, the model is set with a prompt word, requiring the model to output the comprehensive diagnosis of the current collected information. Then, through these comprehensive diagnoses, the possible diseases of the user are analyzed, and then the constitution of the user is judged according to the characteristics of the eight TCM constitutions. The place where the user coincides with the typical characteristics in the eight TCM constitutions is output. Finally, according to the above comprehensive information, the risk of the current user's disease is analyzed, and then the special text generation ability of the large model is used to give targeted suggestions for the current disease.

[0127] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent substitutions for part or all of the technical features; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the scope defined by the claims of the present application.

Claims

1. An intelligent TCM diagnostic system based on an AI large-scale model, characterized in that: It includes an information collection module, a data processing module, a diagnosis generation module, and a large-scale diagnostic model module specific to traditional Chinese medicine; The information collection module collects user information through four parts: facial recognition, tongue image recording, patient self-report, and traditional Chinese medicine palpation. The facial recognition and tongue image recording parts obtain facial and tongue information by recording videos online. The patient self-report part obtains the patient's relevant condition by recording videos online, extracts the voice information from the video, and then obtains text information through voice transcription technology. The traditional Chinese medicine palpation part obtains pulse diagnosis heart rate charts by recording videos online and using a non-contact heart rate measurement system. The data processing module preprocesses the five types of information collected by the information collection module. For facial and tongue information, the data processing module extracts one frame per second from the video. For text information, the data processing module segments the text into words and converts it into numerical form. For pulse and heart rate graphs, the data processing module performs FFT transformation to obtain a spectrogram. For speech information, the data processing module obtains the corresponding speech features through WaveNet. The diagnostic generation module diagnoses the data output from the data processing module. Several frames of images obtained from facial and tongue images are encoded using a visual encoder through a pre-trained VisualGLM model to generate image features. These features are then combined with text prompts, and an autoregressive decoder generates diagnostic results for the face and tongue. The pre-processed text is processed by a pre-trained BERT-over-BERT model, which uses the text to understand the user's role. Pre-defined questions are posed to the BERT-over-BERT model to generate the user's medical history, symptoms, living environment, sweating, diet, sleep patterns, and bowel movements, as well as the presence of headaches, chest pain, and lower back / abdominal pain. The output text is then used to complete the process. Diagnosis: The spectrogram generated from the pulse heart rate map is used to calculate the probability of related cardiovascular diseases by using a heart sound classification method based on a hybrid model CRNN based on CNN and LSTM and a hybrid model ARNN based on time-frequency component attention, and the corresponding pulse diagnosis is output. The speech features generated from the speech information are used by a pre-trained MViTv2 model to output the preliminary classification of the user's voice, including the strength, pitch, clarity, length, and speed of the user's voice. Then, the preliminary classification results are input into a pre-trained SVM model to output the judgment of the patient's lung qi status, emotional changes, qi and blood sufficiency, spleen and stomach diseases, and the health of the internal organs, which are used as the speech diagnosis. Finally, the five diagnoses are input into the ChatGLM4 model to output a summary text to form a comprehensive diagnosis. The TCM-specific large-scale model diagnostic module takes the comprehensive diagnosis generated by the diagnostic generation module as input, and then outputs corresponding treatment suggestions through the pre-trained TCM-specific large-scale model.

2. The intelligent TCM diagnostic system based on an AI large model according to claim 1, characterized in that: The non-contact heart rate measurement system in the information collection module consists of information acquisition hardware and data processing software. The process of acquiring a heart rate graph by the non-contact heart rate measurement system is divided into pulse data acquisition, pulse data processing, heart rate signal processing, and real-time and stable heart rate calculation. The specific steps are as follows: Step 1.1: Pulse data acquisition and processing; The non-contact heart rate measurement system extracts features from the "cun, guan, chi" points by photographing the user's wrist and calculates the average green channel value of the region of interest (ROI). Step 1.2: Heart rate signal processing; Calculate the frames per second (FPS) of the average value of the green channel each time, perform matrix transformation on the average value of the green channel and interpolate it to a uniform time interval, and perform signal processing to obtain the time-division signal of heart rate information; use Fourier transform to convert the time-division signal into a frequency-division signal; Step 1.3: Real-time heart rate and steady-state heart rate calculation; obtain the heart rate spectrum by combining the frequency array calculated by FPS with the amplitude spectrum of the frequency division signal; calculate the real-time heart rate by the peak value of the spectrum within a reasonable range, and then calculate the steady-state heart rate.

3. The intelligent TCM diagnostic system based on an AI large model according to claim 2, characterized in that: In step 1.1, the specific method for pulse data acquisition is as follows: the user first selects to upload a video or record a video in real time according to the guidance, then correctly places the non-contact heart rate measurement system, turns on the built-in light of the non-contact heart rate measurement system, and ensures that the light source is stable. The user is then asked to place their wrist in a designated area of ​​the non-contact heart rate measurement system in the video, maintaining a certain postural stability to ensure that key features around the pulse are fully recorded. After obtaining qualified video data, the non-contact heart rate measurement system will automatically capture a specified number of keyframes from the video stream based on the clarity and completeness of the wrist features it displays. In step 1.1, the specific method for pulse data processing is as follows: The system first identifies the region of interest (ROI) of the wrist, which is described by the following formula: (x, y) → ROI, where (x, y) represents the coordinates of the ROI center. The system marks and divides the wrist region in the image to ensure that subsequent steps only analyze and crop this region. The specific steps are as follows: S1.1.1: Identify the boundary coordinates of the wrist area and divide it into three parts: cun, guan, and chi; S1.1.2: Perform color space conversion and masking on each part separately to ensure that only the skin area is retained; S1.1.3: Find the maximum contour of the skin area and extract the ROI based on it; S1.1.4: For images where the wrist region is successfully identified, the average value of the green channel is further extracted as the basic data for the heart rate signal.

4. The intelligent TCM diagnostic system based on an AI large model according to claim 3, characterized in that: The specific method for step 1.2 is as follows: S1.2.1: Color extraction and standardization; Extract the average value of the green channel in the RGB color channels from the ROI after image processing and perform standardization. The extracted average value is: Where G is the average value of the green channel, g i Let be the green value of the i-th pixel, and N be the total number of pixels; The average value of the green channel is normalized using the following formula: Where Normalized_data is the average value of the green channel after normalization, and G is the average value of the original green channel; S1.2.2: Signal Detrending and Interpolation; After obtaining the average value of the green channel of the pulse ROI, the detected values ​​are stored in a data buffer in the form of a queue. When the buffer is full, the data in the buffer is converted into a matrix signal; then, the signal data is detrending processed; the timestamp of each detection of the average value of the green channel is calculated to obtain the actual FPS, that is, the number of detections per second; based on the periodic time series obtained by the FPS, the average value of the green channel is interpolated into the sequence of average time intervals to obtain the time-division signal of heart rate information; S1.2.3: Fast Fourier Transform; A Fast Fourier Transform is performed on the time-division signal to obtain the frequency-division signal. Then, based on the standard heart rate range of 50 to 180, a bandpass filter is used for spectral analysis to filter out heart rate signals within a specific frequency range. The Fourier Transform formula is as follows: FFT=|F(N t *30)| 2 Where, N t For a normalized time-division signal, F is the Fourier transform operator: Where x(t) is the time-division signal, t is time, X(f) is the frequency domain signal, and f is the signal frequency.

5. The intelligent TCM diagnostic system based on an AI large model according to claim 4, characterized in that: The specific method for step 1.3 is as follows: Based on the transformed frequency-division signal, the frequency array is calculated using the following formula: Where freqs_bpm[k] is the frequency array converted to heartbeats per minute, and k is the frequency index. fps is the sampling frequency, and D is the signal length; Perform spectral analysis on the frequency array to filter out heart rates within the standard range, obtaining a heart rate spectrum. Find the peak frequency in the spectrum, which is the current real-time heart rate. Calculate the average of all historically measured real-time heart rates and then use the following formula to calculate the stable heart rate value over a period of time: Where bpms represents steady-state heart rate, K is the number of historical heart rate data points, and bpm is the mean heart rate. i This is the heart rate value calculated for the i-th time; Ultimately, the system will provide the user with the corresponding heart rate graph, real-time heart rate, and stable heart rate within the "cunguanchi" area.

6. The intelligent TCM diagnostic system based on an AI large model according to claim 1, characterized in that: The facial and tongue diagnosis generation module is divided into a data acquisition section, a data processing section, a VisualGLM fine-tuning section, and a VisualGLM diagnosis section. The specific steps are as follows: Step 2.1: Data Collection; The AI-based intelligent TCM diagnostic system guides users to collect facial or tongue image data via online video by facing the camera, opening their mouth and sticking out their tongue. Step 2.2: Data processing; Using the known YOLOX and the open-source TongueSAM model based on Segment Anything, bounding box cues are generated through an image encoder, cue generator, and mask decoder to achieve high-precision tongue image segmentation, thereby extracting the face and tongue from the acquired video data respectively; Step 2.3: VisualGLM fine-tuning; The system uses the LoRA (Low-Rank Adaptation) method to fine-tune the multimodal dialogue language model VisualGLM. Step 2.4: The processed facial or tongue image, combined with specific prompts, is fed into the VisualGLM model for processing; VisualGLM-6B encodes the input image through an image encoder, and combines it with the prompts generated by the prompt generator to generate descriptions and analysis results of the corresponding tongue or facial images.

7. The intelligent TCM diagnostic system based on an AI large model according to claim 6, characterized in that: In step 2.2, for the face diagnosis module, the system uses YOLOX to extract the face portion of the image as the region of interest (ROI); (x, y) represents the coordinates of the ROI center, and w and h represent the width and height of the ROI, respectively; then the boundary coordinates of the ROI region are determined, with (x1, y1) representing the upper left corner coordinates and (x2, y2) representing the lower right corner coordinates, then... For the tongue diagnosis module, YOLOX is first used to mark the ROI containing the tongue. Then, for images in which the tongue region is successfully identified, the TongueSAM algorithm is used to calculate the probability of the category to which each pixel belongs, and the tongue part in the image is segmented and extracted.

8. The intelligent TCM diagnostic system based on an AI large model according to claim 6, characterized in that: The specific method for step 2.3 is as follows: The system first prepares a training dataset D = {(x i , t i )}, where x i Represents a facial image or a tongue image, t i This represents the corresponding text description; Visual encoder E using the VisualGLM model v Process image data to obtain the image feature representation v i =E v (x i Text encoder E using the VisualGLM model t Process the text description to obtain text features t′ i =E t (t i ); Introducing a low-rank matrix W to update the model's weights, and then applying image features v... i and text features t′ i Combined with the low-rank matrix W, the model output y is calculated. i : y i =f(v i ,t′ i ;θ,W) Where f represents the forward propagation function of the model, y i The model is based on the input v i and t′ i The output prediction results; By minimizing the predicted result y i and real text description t i We use a loss function L to perform supervised learning training, minimizing the difference between the predicted results and the true values: Where I is the loss function, used to measure the difference between the predicted result and the actual text description.

9. The intelligent TCM diagnostic system based on an AI large model according to claim 1, characterized in that: The TCM-specific large-scale diagnostic module uses a pre-trained model to set prompt words. Based on comprehensive diagnostic information, it analyzes the user's possible illnesses, judges the user's constitution based on the characteristics of the eight TCM constitutions, outputs the areas where the user's constitution overlaps with the typical characteristics of the eight TCM constitutions, and the characteristics of the corresponding constitution. Based on the above comprehensive information, it analyzes the current user's possible disease risks and provides corresponding suggestions based on the user's constitution and possible illnesses. The pre-trained model is a large model with TCM-specific knowledge, trained by incorporating the judgment of the eight major constitutions in TCM through a large number of diagnostic cases in the early stage.

Images