Traditional Chinese medicine multi-modal llm training method and application thereof

Abstract

The application provides a traditional Chinese medicine multi-modal LLM training method and application thereof, and relates to the cross field of biology and artificial intelligence. The method comprises the following steps: acquiring multi-modal medical data with labels; training a visual and time sequence feature extraction model to generate tongue fur and pulse features; performing joint features on the tongue fur and pulse features through a cross-modal self-attention mechanism model; training a text feature extraction model to generate text features; inputting the joint features and the text features into the cross-modal attention mechanism model to generate fusion features; and finally integrating each model into a large language model to generate a traditional Chinese medicine multi-modal LLM. The aforementioned training method improves the collaborative expression capability of multi-modal information, enhances the deep semantic association between features, and realizes efficient fusion of traditional Chinese medicine multi-modal information.

Description

TECHNICAL FIELD

[0001] The present application relates to the cross field of biology and artificial intelligence, in particular to a traditional Chinese medicine multi-modal LLM training method and application thereof. BACKGROUND

[0002] Traditional Chinese medicine is the treasure of the Chinese nation, which records the experience and knowledge of Chinese people in fighting against diseases and pursuing health for thousands of years. Creating a feasible traditional Chinese medicine knowledge question and answer method can not only serve the practitioners in the field of traditional Chinese medicine, but also facilitate ordinary users to obtain and understand traditional Chinese medicine knowledge. At present, researchers mostly create traditional Chinese medicine knowledge question and answer methods based on text information. Such methods have some obvious shortcomings. For example, for some traditional Chinese medicine ancient literature, it cannot well cope with the complex language expression methods of various forms such as classical Chinese and colloquial language, resulting in limited semantic understanding and knowledge extraction capability. At the same time, the existing question and answer method relies on a single data source, which cannot comprehensively integrate multiple data types (such as tongue image, pulse image, etc.), resulting in limited judgment accuracy and individualization requirements. SUMMARY

[0003] The present application aims to at least solve at least one of the existing problems. To this end, the present application provides a traditional Chinese medicine multi-modal large language model (LLM) training method, which is used for integrating multi-modal data for training, effectively improving the multi-modal reasoning ability of the model and the final decision accuracy.

[0004] Specifically, the present application provides the following technical solutions:

[0005] In the first aspect of the present application, a traditional Chinese medicine multi-modal LLM training method is provided, which comprises: S1, acquiring labeled traditional Chinese medicine multi-modal medical data, the multi-modal medical data comprising at least one of tongue image, pulse image or text data, the text data comprising at least one of patient information, chief complaint or diagnosis result; S2, training a visual feature extraction model and a time series feature extraction model respectively by using the labeled tongue image and pulse image, to generate tongue visual features and pulse time series features; S3, training a cross-modal self-attention mechanism model by using the tongue visual features and the pulse time series features, to generate joint features; S4, training a text feature extraction model by using the text data, to generate text features; S5, inputting the joint features and the text features into the cross-modal attention mechanism model, to generate fusion features; S6, integrating the trained visual feature extraction model, time series feature extraction model, cross-modal self-attention mechanism model, text feature extraction model and cross-modal attention mechanism model into a large language model, to generate a traditional Chinese medicine auxiliary diagnosis model.

[0006] The model training method of the present application fully excavates the key information in the tongue image, pulse image and text data by training the visual feature extraction model, the time sequence feature extraction model and the text feature extraction model respectively; the cross-modal self-attention mechanism model is used to fuse the tongue visual feature and the pulse time sequence feature, further improving the collaborative expression ability of the multi-modal information; the cross-modal attention mechanism model is used to fuse the joint feature and the text feature, enhancing the deep semantic association between the features. Finally, all the models are integrated into a large language model, realizing efficient fusion of multi-modal information of traditional Chinese medicine.

[0007] In the second aspect of the present application, the present application proposes the application of the traditional Chinese medicine multi-modal LLM in knowledge extraction and relationship extraction of multi-source data, wherein the traditional Chinese medicine multi-modal LLM is trained based on the traditional Chinese medicine multi-modal LLM training method of the first aspect.

[0008] The model trained based on the foregoing traditional Chinese medicine multi-modal LLM training method for knowledge extraction and relationship extraction can effectively improve the comprehensiveness of knowledge extraction and the accuracy of relationship extraction,

[0009] In the third aspect of the present application, the present application proposes the application of the traditional Chinese medicine multi-modal LLM in patient diagnosis, wherein the traditional Chinese medicine multi-modal LLM is trained based on the traditional Chinese medicine multi-modal LLM training method of the first aspect.

[0010] The model trained based on the foregoing traditional Chinese medicine multi-modal LLM training method for patient diagnosis can fully integrate multi-modal data such as tongue image, pulse image and text data, and realize comprehensive analysis and accurate diagnosis of the patient's health condition through the deep reasoning ability of the cross-modal attention mechanism and the large language model. The model uses the joint information of visual, time sequence and text features, not only can efficiently capture the potential association in multi-modal data, but also can provide personalized disease diagnosis and treatment suggestions based on the specific knowledge in the field of traditional Chinese medicine, significantly improve the accuracy, timeliness and intelligent level of diagnosis, thereby reducing the workload of doctors and providing better medical services for patients.

[0011] In the fourth aspect of the present application, the present application proposes the application of the traditional Chinese medicine multi-modal LLM in the teaching task of traditional Chinese medicine, wherein the traditional Chinese medicine multi-modal LLM is trained based on the traditional Chinese medicine multi-modal LLM training method of the first aspect.

[0012] The model trained based on the aforementioned Chinese medicine multi-modal LLM training method can be used for Chinese medicine teaching tasks, can integrate multi-modal information such as tongue coating images, pulse images and related text data, and can comprehensively present Chinese medicine theory and clinical knowledge through cross-modal attention mechanism and knowledge expression ability of the large language model. The model can not only provide intuitive demonstration of multi-modal data, but also generate detailed teaching explanations and case analyses through deep semantic understanding, helping learners deepen their understanding of Chinese medicine theory and practice. At the same time, the model can improve the interactivity and practicality of teaching through interactive question and answer and simulated diagnosis scenarios, providing intelligent, personalized and diversified learning tools for Chinese medicine education, thereby optimizing teaching effect and learning experience.

[0013] In a fifth aspect, the present application provides a computing device. According to embodiments of the present application, the computing device comprises a processor and a memory; the memory is configured to store a computer program; and the processor is configured to execute the computer program to implement the Chinese medicine multi-modal LLM training method according to the first aspect of the present application.

[0014] The aforementioned computing device implements the aforementioned Chinese medicine multi-modal LLM training method through automatic execution of computer instructions, realizes high efficiency and automation, and effectively improves the training efficiency.

[0015] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0017] Figure 1 is a Chinese medicine multi-modal LLM training method flowchart provided by some examples of the present application;

[0018] Figure 2 is a Chinese medicine multi-modal LLM training system schematic diagram provided by some examples of the present application;

[0019] Figure 3 is an electronic device schematic diagram provided by some examples of the present application. DETAILED DESCRIPTION

[0020] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0021] In this document, the terms "first", "second", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In the present application, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices. In the description of the present application, "a plurality of" means two or more than two, unless otherwise specified.

[0022] The present application proposes a traditional Chinese medicine multi-modal LLM training method, a traditional Chinese medicine multi-modal LLM training system, an application of a traditional Chinese medicine multi-modal LLM in knowledge extraction and relationship extraction of multi-source data, an application of a traditional Chinese medicine multi-modal LLM in patient diagnosis, an application of a traditional Chinese medicine multi-modal LLM in traditional Chinese medicine teaching tasks, and an electronic device. The following will be described in detail respectively:

[0023] Traditional Chinese medicine multi-modal LLM training method

[0024] In one aspect of the present application, the present application proposes a traditional Chinese medicine multi-modal LLM training method, referring to Figure 1 The method comprises:

[0025] S1, acquiring labeled traditional Chinese medicine multi-modal medical data, the multi-modal medical data comprising at least one of a tongue coating image, a pulse image or text data, the text data comprising at least one of patient information, chief complaint symptoms or diagnosis results;

[0026] In some examples of the present application, the training data acquisition of the tongue coating image can be obtained from a public database, or can be obtained by means of a medical imaging device. The image data at least includes tongue color, keratinized epithelium (tongue coating) condition of tongue shedding, tongue papilla condition and other information.

[0027] The label of the aforementioned tongue coating image can be selected from the theory of tongue diagnosis in traditional Chinese medicine.

[0028] In some examples of the present application, the pulse image is selected from the pulse data of a one-minute electrocardiogram, which can be obtained from a public database or collected by a medical imaging device.

[0029] The label of the aforementioned pulse image can be selected from the interpretation of TCM pulse.

[0030] In some examples of the present application, the text data includes patient information (such as age, gender, medical history, etc.), chief complaints (such as specific descriptions of headache, fatigue, stomach pain, etc.), and diagnosis results or preliminary assessments given by doctors. The text data can be selected from the TCM-SD TCM field dialectical dataset and the TCM literature question generation dataset.

[0031] To ensure data quality, each piece of multi-modal medical data is annotated by professionals, and the annotation content includes the corresponding disease category, symptom characteristics, and diagnosis basis, ensuring that the multi-modal features can be accurately mapped with actual medical knowledge during model training. In addition, the collection of multi-modal data in the present application covers different populations and disease types to enhance the generalization ability of the model.

[0032] S2, using the labeled tongue image and pulse image, respectively training a visual feature extraction model and a time series feature extraction model to generate tongue visual features and pulse time series features;

[0033] In the S2 step, the labeled tongue image and pulse image will be used to train the visual feature extraction model and the time series feature extraction model, respectively.

[0034] In some examples of the present application, for different modal data inputs, a modal-specific loss function is used to independently train the corresponding sub-model.

[0035] The tongue image is combined with the corresponding TCM tongue theory as annotation data and input into the visual feature extraction model - Vision Transformer (ViT). The ViT model learns to embed images into a high-dimensional space so that visual features with deep semantics can be extracted from it. During training, the cross-entropy loss function is used as the loss function of the visual module to ensure that the model can correctly map between image input and annotated tongue theory knowledge.

[0036] The pulse image combined with the corresponding pulse label optimized ECG signal as labeled data, input to the time sequence feature extraction model-Timesformer. Timesformer can capture the time sequence dynamic information in the pulse image, and extract the time sequence features of the pulse image. In the training process, the dynamic time warping (DTW) and the loss function of the classification task are combined to optimize the performance of the model. DTW loss can help the model align and compare the variability of ECG signals, such as heart rate acceleration or slowing down, so that the model is more robust when extracting pulse features. The calculation formula of DTW is as follows:

[0037]

[0038] wherein, and are two time sequences, is the path of the time sequence pair, is the Euclidean distance of the sequence elements;

[0039] is the predicted sequence of the model, is the real label sequence, and the DTW loss is represented as:

[0040] The loss function of the classification task cross-entropy is as follows:

[0041]

[0042] wherein, C is the number of categories (according to the pulse description category, divided into 24 categories) is the real label (one-hot representation), is the probability predicted by the model.

[0043] The two loss functions are as follows: ; wherein, (0.2~0.5) and (0.5~0.8) are weight parameters.

[0044] The Euclidean distance is used to measure the difference between the two time sequences, and the matching degree of the two signals is calculated based on the optimal alignment path, which promotes the model to learn how to handle the time changes in the ECG signal.

[0045] In addition, the cross-entropy loss function of the classification task will be used for pulse classification, ensuring that the model can accurately predict the pulse category based on the features extracted from the pulse image.

[0046] S3, using the tongue fur visual features and pulse time sequence features, training a cross-modal self-attention mechanism model to generate joint features;

[0047] In the S3 step, the tongue coating visual features and the pulse timing features are respectively mapped to the same feature space after passing through independent feature extraction models (visual feature extraction model and timing feature extraction model). In this step, the self-attention mechanism model is used to weight and fuse the features of the two modalities by calculating the similarity between the visual features and the timing features. Through such alignment, the model can learn the relationship between the visual and timing features and effectively fuse them, thereby improving the multi-modal reasoning ability of the model. However, due to the different sources and properties of visual features and timing features, their feature spaces also differ. Therefore, before inputting these features into the cross-modal self-attention mechanism model, feature embedding is first needed. For visual features, embedding is performed by using Vision Transformer (ViT), which captures global semantic information in images through self-attention mechanisms, converts original pixel-level information into feature vectors with deep semantics, and embeds high-dimensional image data into a unified vector space to generate feature vectors of uniform dimensions; for timing features, ECG timing data learns the dynamic changes of timing data through the self-attention mechanism of Timesformer, enabling it to consider the context information of the time series when processing pulse images, thereby extracting the timing features of the pulse and embedding the data in a unified vector space. Through these processes, visual and timing features are converted into the same dimensional vector space, allowing them to be effectively aligned in subsequent cross-modal fusion.

[0048] The visual features and timing features are input into the cross-modal self-attention mechanism model based on Transformer, and feature fusion is achieved through weighted learning. This model captures the global relationship between visual features and timing features by calculating their similarity, and combines the information of the two modalities through weighting. To ensure stable multi-modal training, the model gradually unfreezes the weights of the single-modal modules in the first two steps during training, allowing the model to gradually adapt to the feature interaction between different modalities. By using a joint loss function to optimize the entire model, not only the quality of feature extraction for each modality is focused on during training, but also the interactive features between modalities are optimized to ensure effective fusion of visual features and timing features in the cross-modal space. Finally, these joint features are further optimized through residual connection and multi-layer perceptron to form cross-modal fusion representation.

[0049] S4, a text feature extraction model is trained using text data to generate text features;

[0050] After the text data is processed by natural language processing techniques, it is input into a text feature extraction model (such as LLaMA, GPT-4, BERT or its variants, etc.). These models will embed the text data into a high-dimensional vector space to generate text feature vectors with rich semantics. In this process, the vocabulary and sentence structure of the text are converted into fixed-dimensional vector representations, and these embedding vectors capture important information and contextual relationships in the text.

[0051] Compared with visual and temporal features, text features usually have more complex representations in semantic space, so in multi-modal learning, text features need to be aligned with other modal features (such as tongue visual features and pulse temporal features). Through cross-modal alignment, the text features are converted into a space that matches the visual and temporal features, allowing them to be effectively fused with other modal features. This process ensures similarity and semantic consistency between modalities through specific alignment algorithms such as co-training or attention-based alignment methods.

[0052] S5, input the joint feature and the text feature into a cross-modal attention mechanism model to generate a fusion feature;

[0053] In step S5, the joint feature and the text feature are fused through cross-modal attention mechanism for training, aiming to realize effective interaction between the two modalities. Specifically, the text feature is first mapped to the joint feature space, so that the model can learn the shared representation of cross-modal features. In this process, the model calculates the similarity weight between the text feature and the joint feature through dot product attention mechanism, and then performs weighted fusion according to the calculated weight to generate the fusion feature.

[0054] To optimize the performance of the entire model, an end-to-end joint training strategy is adopted in this step. During training, multi-cross-modal alignment loss, classification loss and generation task loss are used to jointly optimize the model. Cross-modal alignment loss ensures the consistency of text features and joint features in the shared space, while classification loss and generation task loss help the model accurately generate results in specific tasks (such as diagnosis, education reasoning, etc.). During training, the parameters of all modalities are gradually updated to ensure that the model can achieve optimal performance in multi-modal interaction, so that the network can learn to balance the features of different modalities and ultimately generate accurate diagnosis or education reasoning results.

[0055] S6, integrating the trained visual feature extraction model, the temporal feature extraction model, the cross-modal self-attention mechanism model, the text feature extraction model and the cross-modal attention mechanism model into a large language model to generate a traditional Chinese medicine auxiliary diagnosis model.

[0056] In step S6, in order to further optimize the semantic consistency of cross-modal features and the fusion effect of the model, a contrast learning method is used to improve the feature alignment capability. Specifically, during training, the visual features, time sequence features and text features in the multi-modal data are treated as positive samples, i.e., the corresponding visual, time sequence and text features come from the same instance or different data modalities (modal features) of the same patient. At the same time, the visual features, time sequence features and text features that do not correspond are treated as negative samples, i.e., the data modalities (modal features) from different instances or different patients are paired.

[0057] Through this construction method of positive and negative samples, the goal of the model is to maximize the similarity between positive samples and minimize the similarity between negative samples. In this way, the model can not only learn the feature representation within the modal, but also effectively enhance the feature alignment capability between different modalities through cross-modal contrast learning. Contrast learning helps to ensure that the visual features, time sequence features and text features remain consistent in the same shared space, thereby improving the cross-modal fusion capability of the model.

[0058] The contrast learning process is combined with a joint loss function (including classification loss, regression loss and generation task loss) to jointly optimize the performance of the entire model through weighting. During model training, the parameters of all modalities are gradually updated to achieve end-to-end optimization. After training is completed, the model can achieve the best performance in multi-modal interaction and can accurately generate diagnosis and treatment reasoning or education reasoning results. This cross-modal feature alignment strategy significantly improves the multi-modal reasoning capability of the model and the accuracy of the final decision.

[0059] The Chinese medicine multi-modal LLM trained based on the foregoing method is particularly suitable for Chinese medicine complex text information extraction, can efficiently and accurately identify, analyze and extract information from complex types of Chinese medicine literature and ancient books. By sorting the source text information, a list of relationships such as prescriptions, corresponding diseases, ingredients and ingredient effects can be obtained, and knowledge graph technology can be further introduced to structure the Chinese medicine knowledge system and enhance the reasoning capability of the system itself.

[0060] Chinese medicine multi-modal LLM training system

[0061] On the other hand, the present application proposes a Chinese medicine multi-modal LLM training system, which refers to Figure 2 The system comprises a multi-modal medical data acquisition module 100, a tongue visual feature and pulse time sequence feature extraction module 200, a feature joint module 300, a text feature extraction module 400, a feature fusion module 500 and an integration module 600. Among them,

[0062] 100, a module for acquiring labeled traditional Chinese medicine multi-modal medical data, the multi-modal medical data comprising at least one of a tongue coating image, a pulse image, or text data, the text data comprising at least one of patient information, chief complaint symptoms, or diagnosis results.

[0063] 200, a module for training a visual feature extraction model and a time series feature extraction model respectively using labeled tongue coating images and pulse images, to generate tongue coating visual features and pulse time series features.

[0064] 300, a module for training a cross-modal self-attention mechanism model using the tongue coating visual features and the pulse time series features, to generate joint features.

[0065] 400, a module for training a text feature extraction model using text data, to generate text features.

[0066] 500, a module for inputting the joint features and the text features into a cross-modal attention mechanism model, to generate fusion features.

[0067] 600, a module for integrating the trained visual feature extraction model, the time series feature extraction model, the cross-modal self-attention mechanism model, the text feature extraction model, and the cross-modal attention mechanism model into a large language model, to generate a traditional Chinese medicine auxiliary diagnosis model.

[0068] In some examples of the present application, the aforementioned modules are connected, and the aforementioned connection includes physical connection or network connection.

[0069] Those skilled in the art can understand that the features and advantages described above for the traditional Chinese medicine multi-modal LLM training method also apply to the system described above, and will not be repeated here.

[0070] It should be understood that the system embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, they will not be repeated here. Specifically, Figure 2 The system shown can perform the embodiments of the traditional Chinese medicine multi-modal LLM training method described above, and the operations and / or functions performed by each module in the system correspond to the method embodiments. For the sake of brevity, they will not be repeated here.

[0071] The system of the embodiments of the present application is described above from the perspective of functional modules in combination with the drawings. It should be understood that the functional modules can be implemented in the form of hardware, or in the form of instructions of software, or in the form of a combination of hardware and software modules. Specifically, the steps of the method embodiments in the embodiments of the present application can be completed by the integrated logic circuit of hardware in the processor and / or instructions in the form of software, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processing for execution, or executed by a combination of hardware and software modules in the code processing. Alternatively, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory to complete the steps in the above method embodiments in combination with the hardware thereof.

[0072] Application of traditional Chinese medicine multi-modal LLM in knowledge extraction and relationship extraction of multi-source data

[0073] In another aspect, the application provides an application of the traditional Chinese medicine multi-modal LLM in knowledge extraction and relationship extraction of multi-source data, wherein the traditional Chinese medicine multi-modal LLM is trained based on the foregoing traditional Chinese medicine multi-modal LLM training method.

[0074] The traditional Chinese medicine multi-modal LLM utilizes the structured data in the database to perform fine-tuning training based on the pre-training model, and uses the formatted dictionary form: {“source data”:“structured data”} as training data to allow the traditional Chinese medicine multi-modal LLM to perform supervised fine-tuning training. The fine-tuned traditional Chinese medicine multi-modal LLM can be used as a traditional Chinese medicine multi-modal knowledge extraction LLM, and the purpose is to extract knowledge from newly added unformatted data, extract important and key information from text, ECG electrocardiogram data, and tongue coating image data, and classify and plan these data respectively, and store them in a unified format as a knowledge database for model training iteration.

[0075] Application of traditional Chinese medicine multi-modal LLM in patient diagnosis

[0076] In another aspect, the application provides an application of the traditional Chinese medicine multi-modal LLM in patient diagnosis, wherein the traditional Chinese medicine multi-modal LLM is trained based on the foregoing traditional Chinese medicine multi-modal LLM training method.

[0077] The TCM multi-modal LLM utilizes structured data in the database to fine-tune the pre-trained model, and uses the formatted dictionary form: {“symptom description + ECG + tongue image”: “diagnosis result”} as training data to allow the TCM multi-modal LLM to perform supervised fine-tuning. The fine-tuned TCM multi-modal LLM can be used as a TCM multi-modal QA consultation LLM to perform traditional “inspection, listening, questioning and palpation” in a modern way, taking the patient’s symptom description, electrocardiogram and tongue image data as input, and outputting the patient’s diagnosis and treatment results and opinions in natural language.

[0078] Application of the TCM multi-modal LLM in the TCM teaching task

[0079] In another aspect, the application proposes an application of the TCM multi-modal LLM in the TCM teaching task, which is trained based on the aforementioned TCM multi-modal LLM training method. The TCM multi-modal LLM utilizes structured data in the database to fine-tune the pre-trained model, and uses the formatted dictionary form: {“symptom description + ECG + tongue image”: “feature description + corresponding diagnosis result”} as training data to allow the TCM multi-modal LLM to perform supervised fine-tuning. The fine-tuned TCM multi-modal LLM can be used as a TCM multi-modal teaching LLM to output the feature analysis of the corresponding data and the diagnosis result of the corresponding features in an end-to-end manner by inputting the “simulated patient’s” symptom description, electrocardiogram and tongue image, so as to perform TCM diagnosis and treatment teaching in a more detailed description manner.

[0080] Computing device

[0081] In yet another aspect, the application proposes a computing device. Based on the computing device, the aforementioned TCM multi-modal LLM training method is executed.

[0082] The term “electronic device” is intended to refer to various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The computing device can also refer to various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown in the figures, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0083] As Figure 3As shown, the electronic device 500 includes a computing unit 501 that can perform various appropriate actions and processes in accordance with a computer program stored in a ROM (Read-Only Memory) 502 or a computer program loaded into a RAM (Random Access Memory) 503 from a storage unit 508. Various programs and data required for the operation of the device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An I / O (Input / Output) interface 505 is also connected to the bus 504.

[0084] A plurality of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, and the like; an output unit 507, such as various types of displays, speakers, and the like; a storage unit 508, such as a magnetic disk, an optical disk, and the like; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0085] The computing unit 501 can be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, and the like. The computing unit 501 performs various methods and processes described above, such as the TCM multi-modal LLM training method. For example, in some embodiments, the TCM multi-modal LLM training method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the aforementioned TCM multi-modal LLM training method by any other appropriate means, such as by means of firmware.

[0086] In this application, logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be considered to be ordered listing of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a product of the manufacturing and / or processing. The computer-readable medium can include, but is not limited to: an electronic connection (an electronic device having one or more wires), a portable computer diskette (a magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber device, and a portable compact disc read-only memory (CD ROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via the optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in the computer memory. The various computer-readable storage media described in this specification can represent one or more devices and / or other machine-readable storage media for storing information. The term "machine- readable storage medium" can include, without being limited to, wireless channels and various other media capable of storing, containing, and / or carrying instruction and / or data.

[0087] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, this can be performed using any one or combination of the following technologies, which are all well known in the art: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0088] Those of skill in the art could readily implement the above described example methods with all or a subset of the recited steps or elements. They can be implemented as computer programs or programs of instructions embodied in medium storage devices.

[0089] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0090] It should be understood that various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present disclosure can be executed in parallel, sequentially or in different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which is not limited herein.

[0091] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0092] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments without departing from the principles and purposes of the present application within the scope of the present application.

Claims

1. A method for training a multi-modal LLM for traditional Chinese medicine, characterized in that, Comprising: S1, obtaining labeled traditional Chinese medicine multi-modal medical data, the multi-modal medical data comprising at least one of tongue coating images, pulse images, or text data, the text data comprising at least one of patient information, chief complaints, or diagnosis results; S2, training a visual feature extraction model and a time series feature extraction model respectively using the labeled tongue coating images and pulse images to generate tongue coating visual features and pulse time series features; S3, training a cross-modal self-attention mechanism model using the tongue coating visual features and the pulse time series features to generate joint features; S4, training a text feature extraction model using the text data to generate text features; S5, inputting the joint features and the text features into the cross-modal attention mechanism model to generate fusion features; S6, integrating the trained visual feature extraction model, the time series feature extraction model, the cross-modal self-attention mechanism model, the text feature extraction model, and the cross-modal attention mechanism model into a large language model to generate a traditional Chinese medicine auxiliary diagnosis model; wherein the visual feature extraction model is selected from Vision Transformer or its variants; the time series feature extraction model is Timesformer or its variants; and the text feature extraction model is selected from LLaMA, GPT-4, BERT, or their variants; The S3 step comprises: feature embedding of visual features and time series features; calculating self-attention distribution based on visual features and time series features to capture global relationships between multi-modalities; optimizing the calculation results through residual connection and multi-layer perception to generate joint features.

2. The method of claim 1, wherein, The S5 step comprises: calculating the similarity weight between the text features and the joint features using point multiplication attention mechanism; weighting and superimposing the text features and the joint features according to the weight to generate fusion features.

3. The method of claim 2, wherein, The model is optimized by the following steps: For data input of different modalities, use modal-specific loss functions to independently train corresponding sub-models; For the fused multi-modal features, use a joint loss function including classification loss, regression loss, and generation loss to jointly optimize the entire model.

4. The method of claim 3, wherein, In the training process, the semantic consistency of cross-modal features is optimized by contrastive learning, including: using corresponding visual features, time series features, and text features in multi-modal data as positive samples; using non-corresponding visual features, time series features, and text features as negative samples; maximizing the similarity of positive samples and minimizing the similarity of negative samples to improve the feature alignment capability of the model.

5. Application of a traditional Chinese medicine multi-modal LLM in knowledge extraction and relationship extraction of multi-source data, wherein the traditional Chinese medicine multi-modal LLM is trained based on the method of any one of claims 1-4.

6. Application of a traditional Chinese medicine multi-modal LLM in patient diagnosis, wherein the traditional Chinese medicine multi-modal LLM is trained based on the method of any one of claims 1-4.

7. Application of a traditional Chinese medicine multi-modal LLM in traditional Chinese medicine teaching tasks, wherein the traditional Chinese medicine multi-modal LLM is trained based on the method of any one of claims 1-4.

8. A computing device, comprising: Comprising: a processor and a memory; The memory is configured to store a computer program. The processor is configured to execute the computer program to implement the traditional Chinese medicine multi-modal LLM training method according to any one of claims 1-4.

Images