Intelligent traditional Chinese medicine diagnosis system based on AI large model

Abstract

The invention provides an intelligent traditional Chinese medicine diagnosis system based on an AI large model, and relates to the technical field of traditional Chinese medicine auxiliary diagnosis and treatment. Comprising an information collection module, a data processing module, a generation diagnosis module and a traditional Chinese medicine special large model diagnosis module. The information collection module is used for collecting user information through face recognition, tongue picture recording, patient self-examination and traditional Chinese medicine diagnosis; the data processing module is used for preprocessing the five kinds of collected information; the generation diagnosis module diagnoses the data output by the data processing module; and the traditional Chinese medicine special large model diagnosis module takes the comprehensive diagnosis generated by the generation diagnosis module as input, and outputs corresponding scheme suggestions through a pre-trained traditional Chinese medicine special large model. Five kinds of information collected by the system can be collected on line, a user does not need to go to a designated place or use related machines, remote diagnosis of the whole system is truly achieved on the premise that the functions are achieved, and the problem that a traditional system is single in diagnosis and suggestion is solved by means of the generation capacity of a large model.

Description

Technical Field

[0001] The present invention relates to the technical field of TCM-assisted diagnosis and treatment, and in particular to an intelligent TCM diagnosis system based on an AI big model. Background Art

[0002] Through the information from the four diagnostic methods of "looking, listening, asking and palpating", Chinese medicine can have an overall grasp of the patient's physiological conditions such as the strength and weakness of Qi and blood, cold and heat, deficiency and excess, and summarize the condition. Then, it uses various treatment methods such as Chinese medicine, acupuncture, massage, cupping therapy, diet therapy, etc. to heal the patient as a whole.

[0003] With the development of information technology, TCM diagnosis has gradually been combined with information technology. The telemedicine diagnosis model eliminates the need for offline communication between doctors and patients, which greatly facilitates TCM diagnosis; the TCM diagnosis data analysis system based on artificial intelligence sends suspected disease combinations to doctors for review through the diagnosis mapping model; the intelligent TCM diagnosis system based on deep learning combines the information of the four diagnoses, uses the convolutional neural network deep learning technology training model for pre-training, matches with the local knowledge base and cases, and accurately locates the patient's condition. However, the information technology currently combined with TCM diagnosis has limitations. The number of local TCM knowledge bases is limited, resulting in the condition may not cover all patients' conditions. At the same time, the generated condition diagnosis and corresponding treatment plan are generated by matching existing cases, and the results are relatively simple and lack of specificity.

[0004] The current AI big model technology uses the transformer decoder and is good at generating targeted language descriptions. At the same time, through model fine-tuning and using the local knowledge base as the data set for model fine-tuning, it can generate a corresponding proprietary big model. The learned knowledge can be interconnected to improve the accuracy of diagnosis. At the same time, the output diagnosis can also be customized, which just makes up for the shortcomings of the current traditional Chinese medicine diagnosis system combined with information technology.

[0005] Therefore, it is necessary to develop an intelligent TCM diagnosis system based on AI big model to make up for the shortcomings of the current diagnosis system. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide an intelligent TCM diagnosis system based on an AI big model in view of the deficiencies of the above-mentioned prior art.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] An intelligent TCM diagnosis system based on AI big model, including information collection module, data processing module, diagnosis generation module and TCM proprietary big model diagnosis module;

[0009] The information collection module collects user information through four parts: facial recognition, tongue image recording, patient self-report and TCM palpation. The facial recognition part and tongue image recording part obtain facial information and tongue image information through online video recording. The patient self-report part obtains the patient's relevant condition through online video recording, extracts the voice from the video to obtain voice information, and then obtains text information through voice transcription technology. The TCM palpation part obtains pulse diagnosis heart rate chart by online video recording and using a non-contact heart rate measurement system.

[0010] The data processing module is used to pre-process the five types of information collected by the information collection module; for facial information and tongue information, the data processing module extracts one frame of picture from the video every second; for text information, the data processing module performs word segmentation on the text and converts the text into numerical form; for pulse diagnosis and heart rate graphs, the data processing module converts them through FFT to obtain a spectrum graph; for voice information, the data processing module obtains the corresponding voice features through WaveNet;

[0011] The diagnostic generation module diagnoses the data output by the data processing module; several frames of pictures obtained from the facial information and tongue image information are respectively encoded by the corresponding pre-trained VisualGLM model, and the visual encoder is used to encode the input tongue image or facial image to generate image features, and then these image features are combined with text prompts to generate facial and tongue diagnosis results through the autoregressive decoder; the pre-processed text is passed through the pre-trained BERT-over-BERT (BoB) model, and the text is used as a paragraph to understand the user role, and then the BoB model is asked questions through pre-set questions to generate the user's medical history, symptoms, living environment, sweating, diet, sleep, excretion, whether there is headache, chest pain, waist and abdominal pain, and output text diagnosis; The spectrum generated by the pulse diagnosis heart rate graph is calculated through the hybrid model CRNN based on CNN and LSTM, and the heart sound classification method based on the hybrid model ARNN based on time-frequency component attention, to calculate the possibility of related cardiovascular diseases and output the corresponding pulse diagnosis diagnosis; the speech features generated by the voice information are output through the pre-trained MViTv2 model to output the preliminary classification of the user's voice, including the strength, pitch, clarity, length, and urgency of the user's voice, and then the preliminary classification results are input into the pre-trained SVM model to output the judgment of the patient's lung qi condition, emotional changes, whether the qi and blood are sufficient, whether the spleen and stomach are diseased, and the health of the internal organs as a voice diagnosis; finally, the five diagnoses are input into the ChatGLM4 model, and a summary text is output to form a comprehensive diagnosis;

[0012] The TCM-specific big model diagnosis module takes the comprehensive diagnosis generated by the diagnostic module as input, and then outputs corresponding solution recommendations through the pre-trained TCM-specific big model.

[0013] Further, the non-contact heart rate measuring system in the information collection module is composed of information collection hardware and data processing software; the process of obtaining the heart rate graph by the non-contact heart rate measuring system is divided into a pulse data collection part, a pulse data processing part, a heart rate signal processing part and a real-time heart rate and stable heart rate calculation part, and the specific steps are as follows:

[0014] Step 1.1: Pulse data collection and processing; The non-contact heart rate measurement system takes a picture of the user's wrist, extracts features of the "Cun, Guan, Chi" parts, and calculates the average value of the green channel of the ROI (region of interest);

[0015] Step 1.2: Heart rate signal processing; calculate the frames per second (FPS) of each extraction of the green channel average value, perform matrix transformation on the green channel average value and interpolate it to a uniform time interval, and obtain the time division signal of the heart rate information after signal processing; use Fourier transform to convert the time division signal into a frequency division signal;

[0016] Step 1.3: Calculate the real-time heart rate and the stable heart rate; obtain the heart rate spectrum diagram by calculating the frequency array through FPS and combining it with the amplitude spectrum of the frequency-divided signal; calculate the real-time heart rate through the peak value of the spectrum diagram within a reasonable range, and then calculate the stable heart rate.

[0017] Further, in the step 1.1, the specific method of the pulse data collection part is: the user first selects to upload a video or shoot a video in real time according to the guidance, correctly places the non-contact heart rate measurement system, turns on the built-in lighting of the non-contact heart rate measurement system, and ensures that the light source is stable; then, the user is required to place the wrist in the designated area of ​​the non-contact heart rate measurement system in the video, and maintain a certain posture stability to ensure that the key features around the pulse are fully recorded; after obtaining the video data of qualified quality, the non-contact heart rate measurement system will automatically capture a designated number of key frames from the video stream based on the clarity and completeness of the wrist features displayed;

[0018] In step 1.1, the specific method of the pulse data processing part is as follows: the system first identifies the region of interest of the wrist, and the process is described by the following formula: (x, y) → ROI, where (x, y) represents the coordinates of the center of the ROI; the system marks the wrist area in the image and divides it to ensure that only this area is analyzed and cropped in the subsequent steps. The specific steps are as follows:

[0019] S1.1.1: Identify the boundary coordinates of the wrist area and divide it into three parts: Cun, Guan, and Chi;

[0020] S1.1.2: Perform color space conversion and mask processing on each part separately to ensure that only the skin area is retained;

[0021] S1.1.3: Find the maximum contour of the skin area and extract the ROI based on it;

[0022] S1.1.4: For the image in which the wrist area is successfully identified, the average value of the green channel is further extracted as the basic data of the heart rate signal.

[0023] Furthermore, the specific method of step 1.2 is as follows:

[0024] S1.2.1: Color extraction and standardization: Extract the average value of the green channel in the RGB color channel from the ROI after image processing and perform standardization;

[0025] The average value extracted is:

[0026]

[0027] Among them, G is the average value of the green channel, g i is the green value of the i-th pixel, and N is the total number of pixels;

[0028] The green channel average value is normalized as follows:

[0029]

[0030] Normalized_data is the average value of the green channel after normalization, and G is the original average value of the green channel;

[0031] S1.2.2: Signal detrending and interpolation; after obtaining the green channel average value of the pulse ROI, store each detected value in the data buffer in the form of a queue. When the buffer is full, convert the data in the buffer into a matrix signal; then perform detrending on the signal data; calculate the timestamp of each detection of the green channel average value to obtain the actual FPS, that is, the number of detections per second; based on the periodic time series obtained from the FPS, interpolate the green channel average value into the sequence of the average time interval to obtain the time-division signal of the heart rate information;

[0032] S1.2.3: Fast Fourier transform: Perform fast Fourier transform on the time-division signal to obtain the frequency-division signal. Then, according to the standard range of heart rate: 50 to 180, a bandpass filter is used to perform spectrum analysis to filter out the heart rate signal within a specific frequency range. The Fourier transform formula is as follows:

[0033] FFT=|F(N t *30)| 2

[0034] Among them, N t is the normalized time-division signal, and F is the Fourier transform operator:

[0035]

[0036] Among them, x(t) is the time division signal, t is the time, X(f) is the frequency domain signal, and f is the signal frequency.

[0037] Furthermore, the specific method of step 1.3 is:

[0038] According to the transformed frequency division signal, the frequency array is calculated as follows:

[0039]

[0040] Among them, freqs_bpm[k] is the result of converting the frequency array to the number of heart beats per minute, k is the frequency index, fps is the sampling frequency, D is the length of the signal;

[0041] Perform spectrum analysis on the frequency array, filter out the heart rate within the standard range, and obtain the spectrum of the heart rate; find the peak value of the frequency in the spectrum, which is the current real-time heart rate; average all real-time heart rates measured historically, and calculate the stable heart rate value over a period of time according to the following formula:

[0042]

[0043] Among them, bpms represents the stable heart rate, K is the number of historical heart rate data, bpm i is the heart rate value calculated for the i-th time;

[0044] Finally, the system will provide the user with the corresponding heart rate graph, real-time heart rate and stable heart rate in the "Cun Guan Chi" area.

[0045] Furthermore, the generation of facial diagnosis and tongue diagnosis in the generation diagnosis module is divided into a data acquisition part, a data processing part, a VisualGLM fine-tuning part and a VisualGLM diagnosis part, and the specific steps are as follows:

[0046] Step 2.1: Data collection; the intelligent TCM diagnosis system based on the AI ​​big model guides the user to collect facial or tongue image data through online video by facing the camera, opening the mouth and sticking out the tongue;

[0047] Step 2.2: Data processing: Using the known YOLOX and the open source TongueSAM model based on Segment Anything, the image encoder, prompt generator, and mask decoder are used to generate bounding box prompts to achieve high-precision tongue segmentation, so as to extract the face and tongue in the collected video data respectively;

[0048] Step 2.3: VisualGLM fine-tuning: The system uses the LoRA (Low-Rank Adaptation) method to fine-tune the multimodal dialogue language model VisualGLM model;

[0049] Step 2.4: The processed facial image or tongue image is sent to the VisualGLM model for processing in combination with specific prompts; VisualGLM-6B encodes the input image through the image encoder, and generates a description and analysis result of the corresponding tongue image or facial image in combination with the prompts generated by the prompt generator.

[0050] Furthermore, in step 2.2, for the facial diagnosis module, the system uses YOLOX to extract the face part in the image as the region of interest, i.e., ROI; (x, y) represents the coordinates of the center of the ROI, w and h represent the width and height of the ROI respectively; then the boundary coordinates of the ROI area are determined, (x1, y1) represents the coordinates of the upper left corner, and (x2, y2) represents the coordinates of the lower right corner, then

[0051]

[0052]

[0053] For the tongue diagnosis module, YOLOX is first used to mark the ROI containing the tongue. Then, for the image in which the tongue area is successfully identified, the TongueSAM algorithm is used to calculate the probability of each pixel belonging to a certain category, and the tongue part in the image is segmented and extracted.

[0054] Furthermore, the specific method of step 2.3 is:

[0055] The system first prepares a training data set D = {(x i ,t i )}, where x i represents a facial image or a tongue image, t i Represents the corresponding text description;

[0056] Visual encoder E using the VisualGLM model v Process the image data to obtain the feature representation v of the imagei =E v (x i ); Text encoder E using VisualGLM model t Process the text description to obtain text features t i ′ =E t (t i );

[0057] Introduce the low-rank matrix W to update the weight of the model and transform the image feature v i and text feature t i ′ Combined with the low-rank matrix W, the model output y is calculated i :

[0058] y i =f(v i ,t i ′ ;θ,W)

[0059] Among them, f represents the forward propagation function of the model, y i The model is based on the input v i and t i ′ Output prediction results;

[0060] By minimizing the predicted result y i and real text description i The loss function L between the two is used to perform supervised learning training to minimize the difference between the predicted results and the true values:

[0061]

[0062] Among them, I is the loss function, which is used to measure the difference between the predicted results and the true text description.

[0063] Furthermore, the TCM-specific large-model diagnosis module sets prompt words through a pre-trained model, analyzes the possible conditions of the user based on the comprehensive diagnosis information, judges the user's constitution based on the characteristics of the eight constitutions of TCM, outputs the places where the user overlaps with the typical characteristics of the eight constitutions of TCM, and the characteristics of the corresponding constitution, analyzes the possible condition risks of the current user based on the above comprehensive information, and gives corresponding suggestions based on the user's constitution and possible conditions;

[0064] The pre-trained model is a large model with proprietary knowledge of traditional Chinese medicine that is trained through a large number of diagnostic cases and incorporating judgments on the eight major constitutions in traditional Chinese medicine.

[0065] The beneficial effects of adopting the above technical solution are: the intelligent TCM diagnosis system based on AI big model provided by the present invention can collect all five types of information online, without going to a designated location or using related machines, and the whole system can truly realize remote diagnosis under the premise of ensuring the realization of functions; when processing facial information and tongue image information, the pre-trained VisualGLM model is used respectively, and the input picture can output a more targeted and richer diagnosis; through the pre-trained TCM proprietary big model, the user's constitution can be judged, and the possible condition of the user can be analyzed in a targeted manner, and corresponding suggestions can also be given. The present invention uses the generation ability of the big model to solve the problem of single diagnosis and suggestion of the past system. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A system structure diagram of an intelligent TCM diagnosis system based on an AI big model provided in an embodiment of the present invention;

[0067] Figure 2 A flowchart of pulse diagnosis using a heart rate detection system provided by an embodiment of the present invention;

[0068] Figure 3 A flowchart of facial diagnosis or tongue diagnosis using VisualGLM provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0070] like Figure 1 As shown, the intelligent TCM diagnosis system of this embodiment is divided into four modules: an information collection module, a data processing module, a diagnosis generation module and a TCM-specific large model diagnosis module.

[0071] The information collection module collects user information through four parts: facial recognition, tongue image recording, patient self-report, and TCM palpation. The facial recognition part obtains facial information through online video recording, the tongue image recording part obtains tongue image information through online video recording, the patient self-report part obtains the patient's relevant condition through online video recording, and extracts the voice from the video to obtain voice information, and then obtains text information through voice transcription technology. The TCM palpation part obtains pulse diagnosis and heart rate charts through online videos and a non-contact heart rate measurement system.

[0072] like Figure 2As shown, the pulse diagnosis heart rate diagram is obtained by the non-contact measurement heart rate system consisting of information acquisition hardware and data processing software, which is divided into a pulse data acquisition part, a pulse data processing part, a heart rate signal processing part and a real-time heart rate and stable heart rate calculation part. The specific steps are as follows:

[0073] Step 1.1: Pulse data collection and processing; The non-contact heart rate measurement system takes a photo of the user's wrist, extracts features of the "Cun, Guan, Chi" parts, and calculates the average value of the ROI green channel. The specific method is as follows:

[0074] Pulse data collection part: The user first follows the guidance, chooses to upload a video or shoot a video in real time, correctly places the measuring instrument, and turns on the lighting to ensure that the light source is stable. Subsequently, the user is required to place the wrist in the specified area of ​​the instrument in the video and maintain a certain posture stability to ensure that the key features around the pulse are fully recorded. After obtaining qualified video data, the system will automatically capture a specified number of key frames from the video stream based on the clarity and completeness of the wrist features.

[0075] Pulse data processing part: The system first identifies the region of interest of the wrist. This process can be described by the following formula: (x, y) → ROI, where (x, y) represents the coordinates of the center of the ROI. The system marks the wrist area in the image and divides it to ensure that only this area is analyzed and cropped in the subsequent steps. The specific steps are as follows:

[0076] S1.1.1: Identify the boundary coordinates of the wrist area and divide it into three parts: Cun, Guan, and Chi;

[0077] S1.1.2: Perform color space conversion and mask processing on each part separately to ensure that only the skin area is retained;

[0078] S1.1.3: Find the maximum contour of the skin area and extract the ROI based on it;

[0079] S1.1.4: For images that successfully identify the wrist area, the system will further extract the average value of the green channel as the basic data of the heart rate signal.

[0080] Step 1.2: Heart rate signal processing; calculate the number of frames per second (FPS.framesper second) of each extraction of the green channel average value, perform matrix conversion on the green channel average value and interpolate it to a uniform time interval, and obtain the time-division signal of the heart rate information after signal processing; use Fourier transform to convert the time-division signal into a frequency-division signal. The specific method is as follows:

[0081] S1.2.1. Color extraction and standardization: Extract the average value of the green channel in the RGB color channel from the ROI after image processing and perform standardization.

[0082] Extract the mean:

[0083]

[0084] Among them, G is the average value of the green channel, g i is the green value of the i-th pixel, and N is the total number of pixels.

[0085] Normalize the green channel average:

[0086]

[0087] Among them, Normalized_data is the average value of the green channel after normalization, and G is the original green channel average value.

[0088] S1.2.2, signal detrending and interpolation. After obtaining the green channel average value of the pulse ROI, store each detected value in the data buffer in the form of a queue. When the buffer is full, convert the data in the buffer into a matrix signal. Then detrend the signal data to eliminate the overall trend effect. Calculate the timestamp of each detection of the green channel average value to obtain the actual FPS, that is, the number of detections per second. This is not the FPS of the camera, but the actual detection FPS. According to the periodic time series obtained from the FPS, the green channel average value is interpolated into the sequence of average time intervals, thereby obtaining the time-division signal of the heart rate information.

[0089] S1.2.3, Fast Fourier Transform. Perform fast Fourier transform on the time-division signal to obtain the frequency-division signal. Then, according to the standard range of heart rate: 50 to 180, use a bandpass filter to perform spectrum analysis to filter out the heart rate signal within a specific frequency range. The Fourier transform formula is as follows:

[0090] FFT=|F(N t *30)| 2

[0091] Where N t is the normalized time domain signal, and F is the Fourier transform operator:

[0092]

[0093] Where x(t) is the time-division signal, t is the time, X(f) is the frequency domain signal, and f is the signal frequency.

[0094] Step 1.3: Real-time heart rate and stable heart rate calculation part: Get the heart rate spectrum through FPS calculation frequency array combined with the amplitude spectrum of the frequency division signal; calculate the real-time heart rate through the peak value of the spectrum within a reasonable range, and then calculate the stable heart rate. The specific method is as follows:

[0095] According to the transformed frequency division signal, the frequency array is calculated:

[0096]

[0097] Among them, freqs_bpm[k] is the result of converting the frequency array to beats per minute (BPM), k is the frequency index, fps is the sampling frequency and D is the length of the signal.

[0098] The frequency array is subjected to spectrum analysis, and the heart rates within the standard range (50-180) are screened out, and the spectrum diagram of the heart rate is obtained.

[0099] Then find the peak value of the frequency in the spectrum graph, which is the current real-time heart rate. Then average all the real-time heart rates measured historically to calculate the stable heart rate value over a period of time:

[0100]

[0101] Among them, bpms represents the stable heart rate, K is the number of historical heart rate data, bpm i is the heart rate value calculated for the i-th time.

[0102] Finally, the system will provide the user with the corresponding heart rate graph, real-time heart rate and stable heart rate in the "Cun Guan Chi" area.

[0103] The data processing module is responsible for preprocessing the five types of information collected. For facial information and tongue information, this module extracts one frame of image from the video per second; for text information, this module segmented the text and converted the text into numerical form; for pulse diagnosis and heart rate graphs, this module converted them through FFT to obtain spectrum graphs; for voice information, this module obtained the corresponding voice features through WaveNet.

[0104] The diagnostic generation module diagnoses the data output by the data processing module. Several frames of pictures obtained from facial information and tongue image information are respectively output through the corresponding pre-trained VisualGLM model to output the corresponding facial diagnosis and tongue image diagnosis. In the early stage, the conversation between the doctor and the patient is used as training data to train the BERT-over-BERT (BoB) model, and then the user's text information is input into this pre-trained model to output the text diagnosis. The spectrum generated by the pulse diagnosis heart rate graph is input into the pre-trained hybrid model CRNN based on CNN and LSTM. The heart sound classification method based on the hybrid model ARNN of time-frequency component attention is used to calculate the possibility of having related cardiovascular diseases and output the corresponding pulse diagnosis. The speech features generated by WaveNet are classified by the MViTv2 model and the SVM model to output the speech diagnosis. Finally, the five diagnoses are input into the ChatGLM4 model, and a summary text is output to form a comprehensive diagnosis.

[0105] like Figure 3 As shown, the steps of generating the facial diagnosis and tongue diagnosis by the diagnosis module are as follows:

[0106] Step 2.1: Data Collection. The system first provides detailed instructions through the user interface to guide users on how to start an online video session on their devices, and ensure good lighting conditions, camera positioning, and a clean background.

[0107] For the facial diagnosis module, users need to keep their faces facing the camera and maintain a stable posture to ensure that the key features of the face are fully recorded. For the tongue diagnosis module, users are required to open their mouths and stick out their tongues in the video, maintaining a certain posture stability to ensure that the key features around the tongue surface are fully recorded.

[0108] Once video data of acceptable quality is obtained, the system will automatically capture a specified number of key frames from the video stream based on the clarity and completeness of facial or tongue features.

[0109] Step 2.2: Data processing: YOLOX and the TongueSAM model based on Segment Anything are used to extract the face and tongue from the collected video data.

[0110] For the facial diagnosis module, the system uses YOLOX to extract the face part of the image as the region of interest (ROI). (x, y) represents the coordinates of the center of the ROI, w and h represent the width and height of the ROI respectively. Then determine the boundary coordinates of the ROI area, using (x1, y1) to represent the coordinates of the upper left corner and (x2, y2) to represent the coordinates of the lower right corner. Then we have

[0111]

[0112] For the tongue diagnosis module, similarly, the system first uses YOLOX to mark the ROI containing the tongue. Then, for the image that successfully identifies the tongue area, the system uses the open source TongueSAM algorithm, uses the pre-trained image encoder to encode the input image, generates a bounding box prompt based on target detection, and uses the mask decoder to segment and extract the image to achieve high-precision tongue segmentation.

[0113] Step 2.3: Fine-tune VisualGLM. The LoRA (Low-Rank Adaptation) method is used to fine-tune the VisualGLM model. VisualGLM-6B is a known multimodal conversational language model that encodes the input tongue image or facial image through the visual encoder and combines it with the prompt generated by the prompt generator for multimodal processing and analysis. Specifically:

[0114] The system first prepares a training data set D = {(x i ,t i )}, where x i represents a facial image or a tongue image, t i Represents the corresponding text description.

[0115] In the feature extraction stage, the system uses the visual encoder E of the VisualGLM model v To process the image data, we can get the feature representation v of the image i =E v (x i ). At the same time, the text encoder E t Used to process text descriptions and obtain text features t i ′ =E t (t i ).

[0116] In the LoRA fine-tuning stage, the system introduces a low-rank matrix W to update the model weights and transform the image features v i and text feature t i ′ Combined with the low-rank matrix W, the model output y is calculated i :

[0117] y i =f(v i ,t i ′ ;θ,W)

[0118] Among them, f represents the forward propagation function of the model, y i The model is based on the input v i and t i ′ Output prediction results.

[0119] By minimizing the predicted result y i and real text description i The loss function L between the predicted value and the true value is used for supervised learning training to minimize the error between the predicted value and the true value:

[0120]

[0121] Among them, I is the loss function, which is used to measure the difference between the predicted results and the true text description.

[0122] Step 2.4: VisualGLM diagnosis. The processed facial image or tongue image is sent to the VisualGLM model for processing in combination with a specific prompt. The VisualGLM model outputs the facial or tongue image diagnosis result based on the input image and prompt.

[0123] The system uses the pre-fine-tuned VisualGLM model to analyze the pre-processed face or tongue image and generate specific diagnostic results. First, the system inputs the pre-processed face or tongue image into the VisualGLM model, processes the image data through its visual encoder part, and extracts the visual features of the image. The extracted image features F are then fed into the decoder D of the model. The model combines specific prompts to guide the generation of more targeted diagnostic descriptions. This process is expressed by the following formula:

[0124] Y=D(F,P)

[0125] Among them, Y represents the generated diagnosis description, and P is a specific prompt. Finally, the system integrates and formats the generated diagnosis result Y into a diagnosis report.

[0126] The TCM-specific big model diagnosis module takes the comprehensive diagnosis generated by the diagnostic module as input, and then outputs the corresponding solution suggestions through the pre-trained TCM-specific big model. In the early stage, the TCM-specific big model integrates the judgment of the eight major TCM constitutions through a large number of diagnostic cases, and trains a big model with TCM-specific knowledge. When outputting, set prompt words for the model, requiring the model to output a comprehensive diagnosis of the currently collected information, and then analyze the user's possible condition through these comprehensive diagnoses. Then, based on the characteristics of the eight major TCM constitutions, the user's constitution is judged, and the places where the user overlaps with the typical characteristics of the eight major TCM constitutions are output. Finally, based on the above comprehensive information, the possible disease risk of the current user is analyzed, and then the unique text generation ability of the big model is used to give targeted suggestions for the current condition.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. An intelligent TCM diagnosis system based on AI big model, characterized by: It includes information collection module, data processing module, diagnosis generation module and TCM proprietary large model diagnosis module; The information collection module collects user information through four parts: facial recognition, tongue image recording, patient self-report and TCM palpation. The facial recognition part and tongue image recording part obtain facial information and tongue image information through online video recording. The patient self-report part obtains the patient's relevant condition through online video recording, extracts the voice from the video to obtain voice information, and then obtains text information through voice transcription technology. The TCM palpation part obtains pulse diagnosis heart rate chart by online video recording and using a non-contact heart rate measurement system. The data processing module is used to pre-process the five types of information collected by the information collection module; for facial information and tongue information, the data processing module extracts one frame of image from the video per second; for text information, the data processing module performs word segmentation on the text and converts the text into numerical form; for pulse diagnosis and heart rate graphs, the data processing module converts them through FFT to obtain a spectrum graph; for voice information, the data processing module obtains the corresponding voice features through WaveNet; The diagnostic module generates a diagnosis for the data output by the data processing module. The several frames of images obtained from the facial information and tongue image information are respectively encoded by the corresponding pre-trained VisualGLM model using a visual encoder to encode the input tongue image or facial image to generate image features. These image features are then combined with text prompts to generate facial and tongue diagnosis results through an autoregressive decoder. The pre-processed text is passed through a pre-trained BERT-over-BERT model, which uses the text as a paragraph to understand the user's role. The BERT-over-BERT model is then asked questions using pre-set questions to generate the user's medical history, symptoms, living environment, sweating, diet, sleep, excretion, and whether there is a headache, chest pain, or waist and abdominal pain, and the text is output. Diagnosis; The spectrum generated by the pulse diagnosis heart rate graph is calculated through the hybrid model CRNN based on CNN and LSTM, and the heart sound classification method based on the hybrid model ARNN based on time-frequency component attention, the possibility of having related cardiovascular diseases is calculated, and the corresponding pulse diagnosis is output; the speech features generated by the voice information are output through the pre-trained MViTv2 model to output the preliminary classification of the user's voice, including the strength, pitch, clarity, length, and urgency of the user's voice, and then the preliminary classification results are input into the pre-trained SVM model to output the judgment of the patient's lung qi condition, emotional changes, whether the qi and blood are sufficient, whether the spleen and stomach are diseased, and the health of the internal organs, which are used as voice diagnosis; finally, the five diagnoses are input into the ChatGLM4 model, and a summary text is output to form a comprehensive diagnosis; The TCM-specific big model diagnosis module takes the comprehensive diagnosis generated by the diagnostic module as input, and then outputs corresponding solution recommendations through the pre-trained TCM-specific big model.

2. According to claim 1, an intelligent TCM diagnosis system based on an AI big model is characterized by: The non-contact heart rate measuring system in the information collection module is composed of information collection hardware and data processing software; the process of obtaining the heart rate graph by the non-contact heart rate measuring system is divided into a pulse data collection part, a pulse data processing part, a heart rate signal processing part and a real-time heart rate and stable heart rate calculation part, and the specific steps are as follows: Step 1.1: Pulse data collection and processing; The non-contact heart rate measurement system takes a picture of the user's wrist, extracts features of the "Cun, Guan, Chi" parts, and calculates the green channel average value of the region of interest, i.e., ROI; Step 1.2: Heart rate signal processing; Calculate the FPS of each green channel average value, perform matrix transformation on the green channel average value and interpolate it to a uniform time interval, and obtain the time division signal of the heart rate information after signal processing; use Fourier transform to convert the time division signal into a frequency division signal; Step 1.3: Calculate the real-time heart rate and the stable heart rate; obtain the heart rate spectrum diagram by calculating the frequency array through FPS and combining it with the amplitude spectrum of the frequency-divided signal; calculate the real-time heart rate through the peak value of the spectrum diagram within a reasonable range, and then calculate the stable heart rate.

3. According to claim 2, an intelligent TCM diagnosis system based on an AI big model is characterized by: In the step 1.1, the specific method of the pulse data collection part is: the user first selects to upload a video or shoot a video in real time according to the guidance, correctly places the non-contact heart rate measurement system, turns on the built-in lighting lamp carried by the non-contact heart rate measurement system, and ensures that the light source is stable; Afterwards, the user is asked to place the wrist in the designated area of ​​the non-contact heart rate measurement system in the video and maintain a certain posture stability to ensure that the key features around the pulse are fully recorded; after obtaining qualified video data, the non-contact heart rate measurement system will automatically capture a designated number of key frames from the video stream based on the clarity and completeness of the wrist features; In step 1.1, the specific method of the pulse data processing part is as follows: the system first identifies the region of interest of the wrist, and the process is described by the following formula: (x, y)→ROI, wherein (x, y) represents the coordinates of the center of the ROI; the system marks the wrist area in the image and divides it to ensure that only this area is analyzed and cropped in the subsequent steps. The specific steps are as follows: S1.1.1: Identify the boundary coordinates of the wrist area and divide it into three parts: Cun, Guan, and Chi; S1.1.2: Perform color space conversion and mask processing on each part separately to ensure that only the skin area is retained; S1.1.3: Find the maximum contour of the skin area and extract the ROI based on it; S1.1.4: For the image in which the wrist area is successfully identified, the average value of the green channel is further extracted as the basic data of the heart rate signal.

4. According to claim 3, an intelligent TCM diagnosis system based on an AI big model is characterized by: The specific method of step 1.2 is as follows: S1.2.1: Color extraction and standardization: Extract the average value of the green channel in the RGB color channel from the ROI after image processing and perform standardization; The average value extracted is: Among them, G is the average value of the green channel, g i is the green value of the i-th pixel, and N is the total number of pixels; The green channel average value is normalized as follows: Normalized_data is the average value of the green channel after normalization, and G is the original average value of the green channel; S1.2.2: Signal detrending and interpolation; after obtaining the green channel average value of the pulse ROI, store each detected value in the data buffer in the form of a queue. When the buffer is full, convert the data in the buffer into a matrix signal; then perform detrending on the signal data; calculate the timestamp of each detection of the green channel average value to obtain the actual FPS, that is, the number of detections per second; based on the periodic time series obtained from the FPS, interpolate the green channel average value into the sequence of the average time interval to obtain the time-division signal of the heart rate information; S1.2.3: Fast Fourier transform: Perform fast Fourier transform on the time-division signal to obtain the frequency-division signal. Then, according to the standard range of heart rate: 50 to 180, a bandpass filter is used to perform spectrum analysis to filter out the heart rate signal within a specific frequency range. The Fourier transform formula is as follows: FFT=|F(N t *30)| 2 Among them, N t is the normalized time-division signal, and F is the Fourier transform operator: Among them, x(t) is the time division signal, t is the time, X(f) is the frequency domain signal, and f is the signal frequency.

5. According to claim 4, an intelligent TCM diagnosis system based on an AI big model is characterized by: The specific method of step 1.3 is: According to the transformed frequency division signal, the frequency array is calculated as follows: Among them, freqs_bpm[k] is the result of converting the frequency array to the number of heart beats per minute, k is the frequency index, fps is the sampling frequency, D is the length of the signal; Perform spectrum analysis on the frequency array, filter out the heart rate within the standard range, and obtain the spectrum of the heart rate; find the peak value of the frequency in the spectrum, which is the current real-time heart rate; average all real-time heart rates measured historically, and calculate the stable heart rate value over a period of time according to the following formula: Among them, bpms represents the stable heart rate, K is the number of historical heart rate data, bpm i is the heart rate value calculated for the i-th time; Finally, the system will provide the user with the corresponding heart rate graph, real-time heart rate and stable heart rate in the "Cun Guan Chi" area.

6. According to claim 1, an intelligent TCM diagnosis system based on an AI big model is characterized by: The generation of facial diagnosis and tongue diagnosis in the generation diagnosis module is divided into a data acquisition part, a data processing part, a VisualGLM fine-tuning part and a VisualGLM diagnosis part. The specific steps are as follows: Step 2.1: Data collection; the intelligent TCM diagnosis system based on the AI ​​big model guides the user to collect facial or tongue image data through online video by facing the camera, opening the mouth and sticking out the tongue; Step 2.2: Data processing: Using the known YOLOX and Segment Anything-based open source TongueSAM model, the image encoder, prompt generator, and mask decoder are used to generate bounding box prompts to achieve high-precision tongue segmentation, thereby extracting the face and tongue from the collected video data respectively; Step 2.3: VisualGLM fine-tuning: The system uses the LoRA (Low-Rank Adaptation) method to fine-tune the multimodal dialogue language model VisualGLM model; Step 2.4: The processed facial image or tongue image is sent to the VisualGLM model for processing in combination with specific prompts; VisualGLM-6B encodes the input image through the image encoder, and generates a description and analysis result of the corresponding tongue image or facial image in combination with the prompts generated by the prompt generator.

7. The intelligent TCM diagnosis system based on AI big model according to claim 6 is characterized by: In step 2.2, for the facial diagnosis module, the system uses YOLOX to extract the face part in the image as the region of interest, i.e., ROI; (x, y) represents the coordinates of the center of the ROI, w and h represent the width and height of the ROI respectively; then the boundary coordinates of the ROI area are determined, (x1, y1) represents the coordinates of the upper left corner, and (x2, y2) represents the coordinates of the lower right corner, then For the tongue diagnosis module, YOLOX is first used to mark the ROI containing the tongue. Then, for the image in which the tongue area is successfully identified, the TongueSAM algorithm is used to calculate the probability of each pixel belonging to a certain category, and the tongue part in the image is segmented and extracted.

8. The intelligent TCM diagnosis system based on AI big model according to claim 6 is characterized by: The specific method of step 2.3 is: The system first prepares a training data set D = {(x i , t i )}, where x i represents a facial image or a tongue image, t i Represents the corresponding text description; Visual encoder E using the VisualGLM model v Process the image data to obtain the feature representation v of the image i =E v (x i ); Text encoder E using VisualGLM model t Process the text description to obtain the text feature t′ i =E t (t i ); Introduce the low-rank matrix W to update the weight of the model and transform the image feature v i and text feature t′ i Combined with the low-rank matrix W, the model output y is calculated i : y i =f(v i ,t′ i ;θ,W) Among them, f represents the forward propagation function of the model, y i The model is based on the input v i and t′ i Output prediction results; By minimizing the predicted result y i and real text description i The loss function L between the two is used to perform supervised learning training to minimize the difference between the predicted results and the true values: Among them, I is the loss function, which is used to measure the difference between the predicted results and the true text description.

9. The intelligent TCM diagnosis system based on AI big model according to claim 1 is characterized by: The TCM proprietary large model diagnosis module uses a pre-trained model to set prompt words, analyzes the possible conditions of the user based on the comprehensive diagnosis information, judges the user's constitution based on the characteristics of the eight constitutions of TCM, outputs the overlap between the user and the typical characteristics of the eight constitutions of TCM, and the characteristics of the corresponding constitution, analyzes the possible disease risk of the current user based on the above comprehensive information, and gives corresponding suggestions based on the user's constitution and possible conditions; The pre-trained model is a large model with proprietary knowledge of traditional Chinese medicine that is trained through a large number of diagnostic cases and incorporating judgments on the eight major constitutions in traditional Chinese medicine.

Images