Method and device for processing traditional Chinese medicine large model

Abstract

The invention provides a traditional Chinese medicine large model processing method and device, and the method comprises the steps: obtaining the multi-modal data of a patient, and the multi-modal data comprise image data, sound data, smell data, pulse data and text data; inputting the image data into an image encoder for encoding, and inputting the sound data, the smell data and the pulse data into a signal encoder for encoding; performing pre-training on the traditional Chinese medicine large model by utilizing the text image pre-training data, and performing post-training by utilizing a multi-modal instruction fine-tuning data set; and inputting the multi-modal data into the trained traditional Chinese medicine large model, and outputting a diagnosis result of the patient. A unified signal encoder framework is adopted, unified modeling is carried out on modes such as smell recognition, voice understanding (corresponding to auscultation and diagnosis) and pulse signal analysis (corresponding to cut and diagnosis), the problems of redundant calculation and model expansion caused by a plurality of encoders in a traditional multi-mode method are solved, the diagnosis accuracy is kept, and meanwhile the diagnosis accuracy is improved. And the overall structure and deployment efficiency of the model are obviously optimized.

Description

Technical Field

[0001] This invention relates to the field of traditional Chinese medicine technology, and in particular to a method and apparatus for processing large-scale traditional Chinese medicine models. Background Technology

[0002] Currently, multimodal medical big data models are developing rapidly and are widely used in various scenarios such as clinical diagnosis, medical image analysis, and medical record writing assistance. While improving doctors' work efficiency, they also enable patients to understand their own conditions more in real time and intuitively, which has important application value and social significance.

[0003] However, most mainstream medical models are currently trained and optimized based on the Western medical system, offering limited support for the widely used Traditional Chinese Medicine (TCM) system in China. In real-world applications, TCM has numerous practical uses, but large-scale models with TCM knowledge are scarce, making it difficult to meet clinical needs. To enable large-scale models to provide TCM-related knowledge with high accuracy, they must undergo specialized adaptation and data processing for the TCM field.

[0004] Therefore, systematically collecting all reliable TCM knowledge from the internet and effectively integrating it into a large-scale model has become a crucial step in constructing a TCM multimodal model. Only when the model learns and masters sufficiently comprehensive TCM knowledge can it possess the basic ability to conduct diagnosis. Furthermore, current multimodal models primarily focus on understanding and processing modalities such as text, speech, and images, while the TCM diagnostic system emphasizes the integrated approach of "inspection, auscultation, inquiry, and palpation," posing more complex and refined demands for modality fusion. Existing multimodal technologies typically introduce separate encoders for each modality (e.g., a large-scale ECG model uses an ECG encoder for ECG signals; a large-scale pulse diagnosis model uses a pulse encoder for pulse signals), resulting in a redundant model structure and a significant increase in the number of parameters. Therefore, designing an efficient modality fusion scheme, utilizing signal encoders to understand multiple TCM modalities, and aligning them to a traditional text-image multimodal model has become a core technological challenge for promoting the practical application of TCM multimodal models. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method and apparatus for processing a large model of traditional Chinese medicine, which adopts a unified signal encoder architecture to uniformly model modalities such as odor recognition, speech understanding (corresponding to auscultation and palpation) and pulse signal analysis (corresponding to palpation and palpation), and realizes auscultation and palpation in traditional Chinese medicine through the signal encoder.

[0006] In a first aspect, embodiments of the present invention provide a method for processing a large-scale TCM model, the method comprising: acquiring multimodal patient data, wherein the multimodal data includes: image data, sound data, odor data, pulse data, and text data; inputting the image data into an image encoder for encoding, and inputting the sound data, odor data, and pulse data into a signal encoder for encoding; pre-training the large-scale TCM model using text-image pre-training data, and post-training using a multimodal instruction fine-tuning dataset; inputting the multimodal data into the trained large-scale TCM model, and outputting the patient's diagnostic results.

[0007] In optional embodiments of this application, the above method further includes: constructing a traditional Chinese medicine dataset; wherein the traditional Chinese medicine dataset includes: a traditional Chinese medicine pre-training dataset and a traditional Chinese medicine instruction fine-tuning dataset; and training a large traditional Chinese medicine model based on the traditional Chinese medicine dataset.

[0008] In an optional embodiment of this application, the above-mentioned TCM pre-training dataset includes: a TCM text pre-training dataset; the steps of constructing the TCM dataset include: collecting TCM textbooks and books, scanning the documents of TCM textbooks and books, and extracting the text as a TCM text pre-training dataset; collecting Internet datasets, and using the Internet dataset filtered based on TCM keywords as a TCM text pre-training dataset; and shuffling and organizing the TCM text pre-training dataset into a text corpus.

[0009] In an optional embodiment of this application, the aforementioned TCM pre-training dataset includes: a TCM image and text pre-training dataset; the steps of constructing the TCM dataset include: collecting TCM textbooks and books, and determining the images and their textual explanations of the TCM textbooks and books based on optical symbol recognition technology; collecting internet datasets, and determining the images and their textual explanations of the internet datasets; constructing image and text data pairs as TCM image and text pre-training datasets based on the images and their textual explanations of the TCM textbooks and books and the images and their textual explanations of the internet datasets.

[0010] In an optional embodiment of this application, the above-mentioned TCM pre-training dataset includes: a TCM speech pre-training dataset; the step of constructing the TCM dataset includes: collecting Internet datasets, determining the TCM corpus in the Internet datasets; converting the TCM corpus into audio based on a text-to-speech generative model; and constructing a data pair of audio and TCM corpus as the TCM speech pre-training dataset.

[0011] In an optional embodiment of this application, the above-mentioned TCM instruction fine-tuning dataset includes: a TCM text instruction fine-tuning dataset; the steps of constructing the TCM dataset include: collecting real doctor-patient question-and-answer datasets, filtering TCM question-and-answer pairs from the real doctor-patient question-and-answer datasets based on keywords; collecting medical question-and-answer data, filtering TCM question-and-answer data from the medical question-and-answer data; and using the TCM question-and-answer pairs and TCM question-and-answer data as the TCM text instruction fine-tuning dataset.

[0012] In an optional embodiment of this application, the aforementioned TCM instruction fine-tuning dataset includes: a TCM illustrated instruction fine-tuning dataset; the steps of constructing the TCM dataset include: collecting manually labeled images, training an image classification model based on the manually labeled images to distinguish whether an image is related to TCM; filtering images in TCM illustrated books based on the image classification model, retaining images and their textual descriptions related to TCM; collecting internet datasets, filtering image datasets from the internet datasets; and constructing a TCM illustrated instruction fine-tuning dataset based on TCM-related images and their textual descriptions and the image dataset.

[0013] In an optional embodiment of this application, the above-mentioned TCM instruction fine-tuning dataset includes: a TCM four diagnostic methods instruction fine-tuning dataset; the step of constructing the TCM dataset includes: collecting TCM four diagnostic methods datasets from the Internet, and constructing a TCM four diagnostic methods instruction fine-tuning dataset based on the TCM four diagnostic methods dataset.

[0014] In an optional embodiment of this application, the above-mentioned step of training a large-scale TCM model based on a TCM dataset includes: continuing to pre-train the basic large-scale TCM model based on a TCM pre-training dataset; adding an image encoder and a signal encoder to the large-scale TCM model, and training the large-scale TCM model based on a TCM text and image pre-training dataset and a TCM speech pre-training dataset; converting the TCM instruction fine-tuning dataset into audio signal data of a unified format and inputting it into the large-scale TCM model; and training the large-scale TCM model based on the mixed and shuffled audio signal data.

[0015] Secondly, embodiments of the present invention also provide a processing device for a large-scale TCM model, the device comprising: a multimodal data acquisition module for acquiring multimodal patient data, wherein the multimodal data includes: image data, sound data, odor data, pulse data, and text data; an encoder encoding module for inputting image data into an image encoder for encoding, and inputting sound data, odor data, and pulse data into a signal encoder for encoding; a diagnosis result output module for pre-training the large-scale TCM model using text-image pre-training data, and post-training using a multimodal instruction fine-tuning dataset; inputting the multimodal data into the trained large-scale TCM model, and outputting the patient's diagnosis result.

[0016] The embodiments of the present invention bring the following beneficial effects: This invention provides a method and apparatus for processing a large-scale TCM model. The method acquires multimodal patient data, including image data, sound data, odor data, pulse data, and text data. Image data is input into an image encoder for encoding, while sound, odor, and pulse data are input into a signal encoder for encoding. The large-scale TCM model is pre-trained using text-image pre-training data and post-trained using a multimodal instruction fine-tuning dataset. The multimodal data is then input into the trained large-scale TCM model, outputting the patient's diagnostic results. This approach employs a unified signal encoder architecture to uniformly model modalities such as odor recognition, speech understanding (corresponding to auscultation), and pulse signal analysis (corresponding to palpation), avoiding redundant computation and model bloating issues caused by multiple encoders in traditional multimodal methods. While maintaining diagnostic accuracy, it significantly optimizes the overall model structure and deployment efficiency.

[0017] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0018] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 A schematic diagram of a traditional multimodal large model provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a method for processing a large-scale TCM model, as provided in an embodiment of the present invention; Figure 3 A schematic diagram of a large-scale model of traditional Chinese medicine provided in an embodiment of the present invention; Figure 4 A schematic diagram illustrating the construction of a TCM pre-training dataset provided in an embodiment of the present invention; Figure 5 A schematic diagram illustrating the construction of a TCM instruction fine-tuning dataset provided in an embodiment of the present invention; Figure 6 A flowchart illustrating a method for training a traditional Chinese medicine dataset according to an embodiment of the present invention; Figure 7A schematic diagram of the structure of a processing device for a large-scale TCM model provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Currently, most existing large-scale medical models are trained and optimized based on the Western medicine system, without effective adaptation to traditional Chinese medicine (TCM) scenarios. This results in a lack of TCM-related knowledge storage, weak semantic understanding capabilities, and difficulty in ensuring accuracy and reliability in TCM diagnosis. Furthermore, see also... Figure 1 The diagram illustrates a traditional multimodal large-scale model. Current mainstream multimodal large-scale models generally adopt a technical approach of building an independent encoder for each modality, such as image encoders, speech encoders, and text encoders. This type of architecture relies on a large amount of heterogeneous data for training, which not only increases data preparation costs but also significantly increases the number of model parameters, placing a heavy burden on actual deployment and operation, especially limiting its application in edge devices or low-resource environments.

[0023] Based on this, the present invention provides a method and apparatus for processing a large-scale TCM model, specifically providing a multimodal modeling method for TCM diagnosis and treatment scenarios, involving a large-scale TCM model based on a large-scale TCM training dataset using the four diagnostic methods of observation, auscultation, inquiry, and palpation. On the one hand, by systematically collecting publicly available data from the Internet and authoritative TCM literature, and combining data cleaning, organization, and structuring processes, a high-quality TCM question-and-answer pair dataset is constructed, injecting a rich and credible TCM knowledge system into the model, effectively improving its diagnostic capabilities in the TCM context. On the other hand, a unified signal encoder architecture is adopted to uniformly model modalities such as odor recognition, speech understanding (corresponding to auscultation), and pulse signal analysis (corresponding to palpation), avoiding the redundant computation and model bloating problems caused by multiple encoders in traditional multimodal methods, and significantly optimizing the overall structure and deployment efficiency of the model while maintaining diagnostic accuracy.

[0024] To facilitate understanding of this embodiment, a method for processing a large-scale TCM model disclosed in this embodiment will first be described in detail.

[0025] Example 1: This invention provides a method for processing a large-scale TCM model; see [link / reference]. Figure 2 The flowchart shown illustrates a method for processing a large-scale TCM model. This method includes the following steps: Step S202: Obtain patient multimodal data, including image data, sound data, odor data, pulse data, and text data.

[0026] See also Figure 3 The diagram shown is a schematic of a large-scale TCM model. In this embodiment, multimodal data such as image data, sound data, odor data, pulse data, and text data can be acquired.

[0027] Step S204: Input the image data into the image encoder for encoding, and input the sound data, odor data and pulse data into the signal encoder for encoding.

[0028] like Figure 3 As shown, in this embodiment, image data can be input into an image encoder for encoding, and sound data, odor data, and pulse data can be input into a signal encoder for encoding. Text data does not need to be encoded.

[0029] This embodiment includes an image encoder and a signal encoder. These two encoders can extract the expressive features of the data based on the characteristics of the input modality and abstract the information covered by this modality in advance. This will help large models understand the signals of these modalities.

[0030] Step S206: Pre-train the TCM big data model using text image pre-training data, and post-train using multimodal instruction fine-tuning dataset; input the multimodal data into the trained TCM big data model and output the patient's diagnosis results.

[0031] like Figure 3 As shown, in this embodiment, the TCM big data model can be pre-trained using text image pre-training data, and then post-trained using multimodal instruction fine-tuning dataset; the multimodal data is input into the trained TCM big data model, and the patient's diagnosis results are output.

[0032] This invention provides a method for processing a large-scale TCM model. The method involves acquiring multimodal patient data, including image data, sound data, odor data, pulse data, and text data. Image data is encoded using an image encoder, while sound, odor, and pulse data are encoded using a signal encoder. The large-scale TCM model is pre-trained using text-image pre-training data and post-trained using a multimodal instruction fine-tuning dataset. The multimodal data is then input into the trained large-scale TCM model, outputting the patient's diagnostic results. This method employs a unified signal encoder architecture to uniformly model modalities such as odor recognition, speech understanding (corresponding to auscultation), and pulse signal analysis (corresponding to palpation), avoiding redundant computation and model bloating issues caused by multiple encoders in traditional multimodal methods. While maintaining diagnostic accuracy, it significantly optimizes the overall model structure and deployment efficiency.

[0033] Example 2: This embodiment provides a training method for a large-scale TCM model. This method is implemented based on the above embodiments, focusing on describing the specific steps of training the large-scale TCM model, such as... Figure 3 As shown, in this embodiment, the large-scale TCM model can be trained based on internet TCM corpora, TCM books, and TCM Q&A questions.

[0034] In some embodiments, a traditional Chinese medicine (TCM) dataset can be constructed; wherein the TCM dataset includes: a TCM pre-training dataset and a TCM instruction fine-tuning dataset; and a large TCM model is trained based on the TCM dataset.

[0035] The construction of the TCM dataset mainly includes the construction of the TCM pre-training dataset and the TCM instruction fine-tuning dataset. The TCM pre-training dataset requires a very large amount of data, while the TCM instruction fine-tuning dataset needs to be presented in the form of question-and-answer data pairs. The specific process is as follows.

[0036] In some embodiments, the above-mentioned TCM pre-training dataset includes: a TCM text pre-training dataset; TCM textbooks and books can be collected, and the documents of TCM textbooks and books can be scanned to extract text as a TCM text pre-training dataset; Internet datasets can be collected, and the Internet dataset filtered based on TCM keywords can be used as a TCM text pre-training dataset; the TCM text pre-training dataset can be shuffled and organized into a text corpus.

[0037] See also Figure 4 The diagram shown illustrates the construction of a TCM pre-training dataset. The main steps in constructing a TCM text pre-training dataset include: Step 1: Collect TCM textbooks and books. After deduplication and filtering out books of low quality, this example collected over 3000 TCM textbooks and books. Then, image-to-text technology was used to scan the PDF files of all books, extracting the text to form part of the TCM text pre-training dataset. The image-to-text technology can be OCR (Optical Character Recognition) technology.

[0038] Step Two: Collect internet datasets (e.g., Common Crawl dataset, WeChat official account article data, etc.). Then, with the doctor's assistance and advice, 300 TCM-related keywords can be identified. These keywords are then used to filter the internet dataset, retaining only the data containing those keywords, ultimately creating a dataset of internet TCM-related corpus.

[0039] Step 3: Integrate the datasets from the first two steps, shuffle and organize them into text corpus for training of the TCM large model.

[0040] In some embodiments, the above-mentioned TCM pre-training dataset includes: a TCM image and text pre-training dataset; it can collect TCM textbooks and books, and determine the images and their textual explanations based on optical symbol recognition technology; it can collect Internet datasets, and determine the images and their textual explanations in the Internet datasets; it can construct image and text data pairs based on the images and their textual explanations in TCM textbooks and books and the images and their textual explanations in the Internet datasets as a TCM image and text pre-training dataset.

[0041] like Figure 4 As shown, the main steps in constructing a TCM image and text pre-training dataset include: Step 1: Locate all images in traditional Chinese medicine books, and use OCR transcription technology to merge and preserve the images and their corresponding text explanations.

[0042] Step 2: Extract all images appearing in the official account and find their corresponding text explanations.

[0043] Step 3: Integrate all the image and text information from the first two steps to construct a trainable "image-text" pair.

[0044] In some embodiments, the above-mentioned TCM pre-training dataset includes: a TCM speech pre-training dataset; internet datasets can be collected to determine TCM corpus from the internet datasets; the TCM corpus is converted into audio based on a text-to-speech generative model; and data pairs of audio and TCM corpus are constructed as the TCM speech pre-training dataset.

[0045] like Figure 4 As shown, the main steps in constructing a TCM speech pre-training dataset include: Step 1: Use keyword filtering strategies to filter TCM-related corpora from the Huatuo26M dataset.

[0046] Step 2: Use a text-to-speech generative model to convert the corpus into audio. The speech generative model can be a TTS (Text to Speech) model.

[0047] Step 3: Construct audio and original text into "audio-text" data pairs for the model to perform speech-text alignment training.

[0048] In some embodiments, the above-mentioned TCM instruction fine-tuning dataset includes: a TCM text instruction fine-tuning dataset; it can collect real doctor-patient question-and-answer datasets, and filter TCM question-and-answer pairs from the real doctor-patient question-and-answer datasets based on keywords; it can collect medical question-and-answer data, and filter TCM question-and-answer data from the medical question-and-answer data; and it can use TCM question-and-answer pairs and TCM question-and-answer data as a TCM text instruction fine-tuning dataset.

[0049] See also Figure 5 The diagram shown illustrates the construction of a TCM instruction fine-tuning dataset. The main steps in constructing a TCM text instruction fine-tuning dataset include: Step 1: Collect a real doctor-patient question and answer dataset, and use keywords to filter out question and answer pairs related to traditional Chinese medicine.

[0050] Step 2: Filter the TCM-related parts from verifiable medical Q&A data.

[0051] Step 3: Integrate the datasets from the first two steps into a text instruction fine-tuning dataset.

[0052] In some embodiments, the above-mentioned TCM instruction fine-tuning dataset includes: a TCM illustrated instruction fine-tuning dataset; manually labeled images can be collected, and an image classification model can be trained based on the manually labeled images to distinguish whether the images are related to TCM; the images in TCM illustrated books can be filtered based on the image classification model to retain images and their text descriptions related to TCM; an internet dataset can be collected, and the image dataset of the internet dataset can be filtered; and a TCM illustrated instruction fine-tuning dataset can be constructed based on TCM-related images and their text descriptions and the image dataset.

[0053] like Figure 5 As shown, the main steps in constructing a TCM graphic instruction fine-tuning dataset include: Step 1: Manually label parts of the image and use the labeled images to train an image classification model to determine whether the input image is related to Traditional Chinese Medicine. The image classification model can be a CLIP (Contrastive Language-Image Pre-training) model.

[0054] Step 2: Use the trained CLIP model to filter the images in the TCM illustrated book, retaining the TCM-related image portions and their corresponding text descriptions.

[0055] Step 3: Filter the image datasets in the Internet dataset (WeChat official account).

[0056] Step 4: Integrate the data from Step 2 and Step 3 to construct a TCM instruction fine-tuning dataset of "image-question-answer".

[0057] In some embodiments, the above-mentioned TCM instruction fine-tuning dataset includes: a TCM four diagnostic methods instruction fine-tuning dataset; TCM four diagnostic methods datasets can be collected from the Internet, and a TCM four diagnostic methods instruction fine-tuning dataset can be constructed based on the TCM four diagnostic methods dataset.

[0058] like Figure 5 As shown, the main steps in constructing the TCM four diagnostic instructions fine-tuning dataset include: Step 1: Collect existing datasets of TCM four diagnostic methods from the internet, such as "pulse - whether pregnant", "odor - what kind of lung disease the patient has", "pulse - whether the patient has a certain disease", "cough sound - whether the patient has pneumonia", etc.

[0059] Step 2: Organize all data into “Traditional Chinese Medicine Special Modality Signals - Problems Related to This Modality - Natural Language Description Annotation Results”, and construct a multimodal instruction fine-tuning training dataset.

[0060] The method provided in this embodiment of the invention can construct a TCM pre-training dataset and a TCM instruction fine-tuning dataset as a TCM dataset. By systematically collecting publicly available data from the Internet and authoritative TCM literature, and combining data cleaning, sorting, and structuring processes, a high-quality TCM question-and-answer pair dataset is constructed, which injects a rich and credible TCM knowledge system into the model and effectively improves its ability to conduct consultations in the TCM context.

[0061] After constructing a TCM pre-training dataset and a TCM instruction fine-tuning dataset as the TCM dataset, this embodiment requires training a large TCM model based on the TCM dataset. (See also...) Figure 6 The flowchart shown illustrates a training method for a traditional Chinese medicine dataset. The training method for this large-scale traditional Chinese medicine model includes the following steps: Step S602: Continue pre-training the basic TCM large model based on the TCM pre-training dataset.

[0062] This embodiment can further pre-train the model based on the basic TCM large model (such as the Qwen2.5-Base model) using the constructed "TCM pre-training dataset (TCM text pre-training dataset)" to enhance the model's textual knowledge reserves in the field of TCM.

[0063] Step S604: Add an image encoder and a signal encoder to the TCM large model, and train the TCM large model based on the TCM image and text pre-training dataset and the TCM speech pre-training dataset.

[0064] This embodiment can add a pre-trained visual encoder (i.e., an image encoder) and a visual adapter to the large-scale TCM model; and add a pre-trained signal encoder (SignalEncoder) and a signal adapter (SignalAdapter) to the large-scale TCM model.

[0065] This embodiment can use the "Traditional Chinese Medicine Pre-training Dataset (Traditional Chinese Medicine Text and Image Pre-training Dataset)" and the "Traditional Chinese Medicine Pre-training Dataset (Traditional Chinese Medicine Speech Pre-training Dataset)" to train the model, endow it with the ability to understand images and speech, and realize multimodal interaction.

[0066] Step S606: Convert the TCM instruction fine-tuning dataset into audio signal data in a unified format and input it into the TCM large model.

[0067] This embodiment unifies the signal format, enabling efficient training. All TCM signal data is converted into audio signal data of a unified format. For example, time-series signal data such as pulse diagnosis data and odor data are transformed into fixed-frequency audio signals. All TCM-specific modalities are input into the large TCM model in audio form, allowing the signal encoder to directly understand these modalities. This approach avoids introducing additional encoders (e.g., eliminating the need to create separate encoders for pulse diagnosis and odor signals), achieving efficient adaptation to TCM-specific modalities.

[0068] The steps for converting signals such as pulse to audio are as follows: First, read the time sequence values ​​and corresponding timestamps of the signal data. Next, after removing duplicate time points, use linear interpolation to expand the sparse signal into a continuous time series, and generate a uniform time grid according to an audio sampling rate of 44.1kHz. Then, center the signal and perform amplitude scaling, normalizing it to the range [-0.9, 0.9] to prevent audio distortion. Finally, convert the signal to a 16-bit integer format and save it as a WAV audio file, thus achieving the conversion from the original signal to an audio file.

[0069] Step S608: Train the TCM large model based on the mixed and shuffled audio signal data.

[0070] In this embodiment, after the audio signal data conversion is completed, all TCM instruction fine-tuning data are mixed and shuffled to train a large TCM model, enabling the large TCM model to understand and apply multiple TCM modalities.

[0071] In this embodiment, traditional Chinese medicine (TCM) examinations can be used as the text evaluation benchmark, and images not found in the training set can be selected from TCM illustrated books to construct an image evaluation benchmark to evaluate the model's performance. The TCM large-scale model outperformed most models, achieving the best results at the same parameter level. This demonstrates that the TCM training dataset proposed in this embodiment is of high quality and covers a wide range of knowledge.

[0072] The training method for the TCM large-scale model proposed in this embodiment enables the model to understand modalities it has encountered before. (For example, in determining whether a patient is pregnant based on a pulse, the accuracy can be improved from 50% to 80.5% based on random selection), achieving understanding of multiple modalities. Existing multimodal large-scale models cannot understand these TCM modalities; this embodiment is the first model to achieve this functionality.

[0073] The large-scale TCM model provided in this embodiment also features a simple model structure. In the prior art, if multiple modalities need to be added, multiple encoders need to be adapted for each modality. Each modality has its own encoder and adapter. If more modalities are needed, even more encoders and adapters are required, resulting in an overly complex model structure.

[0074] The large-scale TCM model provided in this embodiment can understand special TCM modalities (pulse signals, odor signals, and sound signals) through a unified signal encoder, which can reduce the number of model parameters and avoid overly complex structures.

[0075] Example 3: Corresponding to the above method embodiments, this invention provides a processing device for a large-scale TCM model, see [link to relevant documentation]. Figure 7 The diagram shows a structural schematic of a processing device for a large-scale TCM model. The processing device includes: The multimodal data acquisition module 71 is used to acquire patient multimodal data, including image data, sound data, odor data, pulse data, and text data. The encoder encoding module 72 is used to input image data into the image encoder for encoding, and to input sound data, odor data and pulse data into the signal encoder for encoding; The diagnosis result output module 73 is used to pre-train the TCM big model using text image pre-training data and post-train using multimodal instruction fine-tuning dataset; it inputs multimodal data into the trained TCM big model and outputs the patient's diagnosis result.

[0076] This invention provides a processing device for a large-scale TCM model, which acquires multimodal patient data, including image data, sound data, odor data, pulse data, and text data. Image data is input into an image encoder for encoding, while sound, odor, and pulse data are input into a signal encoder for encoding. The large-scale TCM model is pre-trained using text-image pre-training data and post-trained using a multimodal instruction fine-tuning dataset. The multimodal data is then input into the trained large-scale TCM model, outputting the patient's diagnostic results. This method employs a unified signal encoder architecture to uniformly model modalities such as odor recognition, speech understanding (corresponding to auscultation), and pulse signal analysis (corresponding to palpation), avoiding redundant computation and model bloating problems caused by multiple encoders in traditional multimodal methods. While maintaining diagnostic accuracy, it significantly optimizes the overall model structure and deployment efficiency.

[0077] The aforementioned device also includes: a large-scale TCM model training module for constructing a TCM dataset; wherein the TCM dataset includes: a TCM pre-training dataset and a TCM instruction fine-tuning dataset; and a large-scale TCM model is trained based on the TCM dataset.

[0078] The aforementioned TCM pre-training dataset includes: a TCM text pre-training dataset; the aforementioned TCM large-scale model training module, which collects TCM textbooks and books, scans the documents of TCM textbooks and books, extracts the text as the TCM text pre-training dataset; collects internet datasets, and uses the internet dataset filtered based on TCM keywords as the TCM text pre-training dataset; and shuffles and organizes the TCM text pre-training dataset into text corpus.

[0079] The aforementioned TCM pre-training dataset includes: a TCM image and text pre-training dataset; the aforementioned TCM large-scale model training module, used to collect TCM textbooks and books, and determine the images and their textual interpretations based on optical symbol recognition technology; collect internet datasets, and determine the images and their textual interpretations based on the internet datasets; and construct image-text data pairs based on the images and their textual interpretations from TCM textbooks and books and the images and their textual interpretations from the internet datasets as the TCM image and text pre-training dataset.

[0080] The aforementioned TCM pre-training dataset includes: a TCM speech pre-training dataset; the aforementioned TCM large-scale model training module, used to collect internet datasets and determine the TCM corpus within the internet datasets; converting the TCM corpus into audio based on a text-to-speech generative model; and constructing data pairs of audio and TCM corpus as the TCM speech pre-training dataset.

[0081] The aforementioned TCM instruction fine-tuning dataset includes: a TCM text instruction fine-tuning dataset; the aforementioned TCM large model training module, used to collect real doctor-patient question-and-answer datasets, and to filter TCM question-and-answer pairs from real doctor-patient question-and-answer datasets based on keywords; to collect medical question-and-answer data, and to filter TCM question-and-answer data from medical question-and-answer data; and to use TCM question-and-answer pairs and TCM question-and-answer data as the TCM text instruction fine-tuning dataset.

[0082] The aforementioned TCM instruction fine-tuning dataset includes: a TCM illustrated instruction fine-tuning dataset; the aforementioned TCM large-scale model training module, used to collect manually labeled images and train an image classification model based on these images to distinguish whether an image is related to TCM; filtering images in TCM illustrated books based on the image classification model, retaining images and their textual descriptions related to TCM; collecting internet datasets and filtering image datasets from the internet datasets; and constructing a TCM illustrated instruction fine-tuning dataset based on TCM-related images, their textual descriptions, and the image dataset.

[0083] The aforementioned TCM instruction fine-tuning dataset includes: a TCM four diagnostic methods instruction fine-tuning dataset; and the aforementioned TCM large model training module, which is used to collect TCM four diagnostic methods datasets from the Internet and construct a TCM four diagnostic methods instruction fine-tuning dataset based on the TCM four diagnostic methods dataset.

[0084] The aforementioned TCM large-scale model training module is used to further pre-train the basic TCM large-scale model based on the TCM pre-training dataset; add an image encoder and a signal encoder to the TCM large-scale model; train the TCM large-scale model based on the TCM text and image pre-training dataset and the TCM speech pre-training dataset; convert the TCM instruction fine-tuning dataset into audio signal data in a unified format and input it into the TCM large-scale model; and train the TCM large-scale model based on the mixed and shuffled audio signal data.

[0085] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the TCM large-scale model processing device described above can be referred to the corresponding process in the aforementioned TCM large-scale model processing method embodiments, and will not be repeated here.

[0086] Example 4: This invention also provides an electronic device for running the above-mentioned processing method of the large-scale TCM model; see [link to related document]. Figure 8 The diagram shows the structure of an electronic device, which includes a memory 100 and a processor 101. The memory 100 is used to store one or more computer instructions, which are executed by the processor 101 to implement the above-mentioned processing method of the TCM large model.

[0087] Furthermore, Figure 8The electronic device shown also includes a bus 102 and a communication interface 103, with the processor 101, the communication interface 103 and the memory 100 connected via the bus 102.

[0088] The memory 100 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 103 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 102 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0089] Processor 101 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 101 or by instructions in software form. Processor 101 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 100, and processor 101 reads information from memory 100 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0090] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the above-described processing method for the TCM large model. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0091] The computer program product of the TCM large model processing method and apparatus provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0092] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and / or device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0093] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0094] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0096] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for processing a large-scale TCM model, characterized in that, The method includes: Acquire patient multimodal data, wherein the multimodal data includes: image data, sound data, odor data, pulse data, and text data; The image data is input into an image encoder for encoding, and the sound data, odor data, and pulse data are input into a signal encoder for encoding. The TCM model is pre-trained using text image pre-training data, and then post-trained using a multimodal instruction fine-tuning dataset. The multimodal data is then input into the fully trained TCM model to output the patient's diagnosis results.

2. The method according to claim 1, characterized in that, The method further includes: Construct a Traditional Chinese Medicine (TCM) dataset; wherein, the TCM dataset includes: a TCM pre-training dataset and a TCM instruction fine-tuning dataset; The large-scale TCM model is trained based on the TCM dataset.

3. The method according to claim 2, characterized in that, The TCM pre-training dataset includes: a TCM text pre-training dataset; the steps for constructing the TCM dataset include: Collect TCM textbooks and books, scan the files of the TCM textbooks and books, and extract the text as the TCM text pre-training dataset; Collect internet datasets and use the internet datasets filtered based on TCM keywords as the TCM text pre-training dataset; The TCM text pre-training dataset was shuffled and organized into a text corpus.

4. The method according to claim 2, characterized in that, The TCM pre-training dataset includes: a TCM image and text pre-training dataset; the steps for constructing the TCM dataset include: Collect TCM textbooks and books, and determine the images and textual explanations of the TCM textbooks and books based on optical symbol recognition technology; Collect internet datasets and determine the images and their textual interpretations within those datasets. Based on the images and textual explanations of the aforementioned TCM textbooks and books, and the images and textual explanations of the aforementioned internet datasets, image-text data pairs are constructed as the TCM image-text pre-training dataset.

5. The method according to claim 2, characterized in that, The TCM pre-training dataset includes: a TCM speech pre-training dataset; the steps for constructing the TCM dataset include: Collect internet datasets and determine the TCM corpus within those datasets; The TCM corpus is converted into audio based on a text-to-speech generative model; Construct data pairs between audio and the TCM corpus as the TCM speech pre-training dataset.

6. The method according to claim 2, characterized in that, The TCM instruction fine-tuning dataset includes: a TCM text instruction fine-tuning dataset; the steps for constructing the TCM dataset include: Collect a real doctor-patient question and answer dataset, and filter the TCM question and answer pairs in the real doctor-patient question and answer dataset based on keywords; Collect medical Q&A data, and then filter traditional Chinese medicine Q&A data from the medical Q&A data; The TCM question-and-answer pairs and the TCM question-and-answer data are used as the TCM text instruction fine-tuning dataset.

7. The method according to claim 2, characterized in that, The TCM instruction fine-tuning dataset includes: a TCM text and image instruction fine-tuning dataset; the steps for constructing the TCM dataset include: Collect manually labeled images, and train an image classification model based on the manually labeled images to distinguish whether the images are related to traditional Chinese medicine. Based on the image classification model, images in traditional Chinese medicine illustrated books are filtered to retain images related to traditional Chinese medicine and their text descriptions. Collect internet datasets and filter the image datasets within those internet datasets; The TCM-related images and their textual descriptions, along with the image dataset, are used to construct the TCM graphic instruction fine-tuning dataset.

8. The method according to claim 2, characterized in that, The TCM instruction fine-tuning dataset includes: a TCM four diagnostic methods instruction fine-tuning dataset; the steps for constructing the TCM dataset include: Collect a dataset of TCM four diagnostic methods from the Internet, and construct a fine-tuning dataset of TCM four diagnostic instructions based on the dataset.

9. The method according to claim 2, characterized in that, The steps for training the large-scale TCM model based on the TCM dataset include: The basic TCM large model is further pre-trained based on the TCM pre-training dataset. An image encoder and a signal encoder are added to the TCM big data model, and the TCM big data model is trained based on the TCM image and text pre-training dataset and the TCM speech pre-training dataset. The TCM instruction fine-tuning dataset is converted into audio signal data in a unified format and input into the TCM large model; The TCM big data model is trained based on the mixed and shuffled audio signal data.

10. A processing device for a large-scale traditional Chinese medicine model, characterized in that, The device includes: A multimodal data acquisition module is used to acquire patient multimodal data, wherein the multimodal data includes: image data, sound data, odor data, pulse data, and text data; The encoder encoding module is used to input the image data into the image encoder for encoding, and to input the sound data, the odor data, and the pulse data into the signal encoder for encoding; The diagnostic result output module is used to pre-train the TCM big data model using text image pre-training data, and post-train using multimodal instruction fine-tuning dataset; the multimodal data is input into the trained TCM big data model, and the diagnostic result of the patient is output.

Images