Training method and device for traditional Chinese medicine auxiliary diagnosis large model and electronic equipment

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a traditional Chinese medicine auxiliary diagnosis large model training method and device and electronic equipment. And in combination with symptom feature information of abnormal items in the traditional Chinese medicine theory as a mark, performing fine tuning training on the pre-trained multi-modal large model. In the process, multi-modal image data and traditional Chinese medicine theoretical knowledge are effectively fused, so that a second multi-modal large model special for traditional Chinese medicine auxiliary diagnosis is generated. According to the model, abnormal item identification can be carried out by accurately applying the traditional Chinese medicine theory based on the face, tongue and face infrared images of the user, so that the intelligent level and accuracy of traditional Chinese medicine identification in online inquiry are remarkably improved, the problems of information transmission limitation and non-uniform doctor resource distribution in the online traditional Chinese medicine inquiry are effectively solved, and the user experience is improved. And the online inquiry efficiency and the medical experience of the patient are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a training method, device and electronic equipment for a large model of traditional Chinese medicine-assisted diagnosis. Background Art

[0002] In the field of traditional Chinese medicine, the traditional four-diagnosis method of "looking, smelling, asking and feeling" is the core means of diagnosing diseases in traditional Chinese medicine. Among them, "asking" is an important way to obtain patients' subjective symptoms and plays an irreplaceable role in clinical diagnosis of traditional Chinese medicine. However, with the rapid development of Internet technology, online medical care has gradually become one of the important ways to solve the problems of uneven distribution of medical resources and difficulty in seeking medical treatment for patients. Especially in the field of traditional Chinese medicine, online consultation is a modern extension of "asking", aiming to allow patients to conveniently obtain traditional Chinese medicine diagnosis and treatment services through remote means.

[0003] Although online consultation has shown great potential in improving the accessibility of medical services, the current online consultation method based on text or voice description still has significant limitations. First, the lack of text description and non-professionalism are one of the main factors restricting the accuracy of diagnosis. Patients often find it difficult to describe their symptoms in accurate and comprehensive Chinese medicine terms, which leads to deviations in the process of information transmission and even misleads the doctor's diagnostic thinking. This information asymmetry not only increases the risk of misdiagnosis, but also reduces patients' medical experience and trust.

[0004] Secondly, although video calls make up for the shortcomings of text descriptions to a certain extent, by intuitively displaying information about the patient's face and other parts of the body, it provides doctors with richer diagnostic evidence. However, this model also faces challenges: on the one hand, the scarcity of high-end medical resources makes it difficult for qualified Chinese medicine experts to take time out to participate in online diagnosis and treatment, making it difficult for online consultation platforms to attract and retain high-quality doctor resources; on the other hand, even if video is used, if the doctor is inexperienced and inexperienced, he may still make a one-sided diagnosis due to the inability to accurately identify the nature of the disease, affecting the treatment effect.

[0005] It can be seen that the online TCM consultation service in the existing technology has limitations in information transmission and uneven distribution of doctor resources, resulting in low consultation efficiency. Summary of the invention

[0006] The purpose of this application is to provide a training method, device and electronic equipment for a large model of TCM-assisted diagnosis, so as to solve the problems of low consultation efficiency caused by the limitations of information transmission and uneven distribution of doctor resources in the online TCM consultation service in the prior art.

[0007] In order to achieve the above purpose, the technical solution adopted in this application is as follows:

[0008] According to one aspect of an embodiment of the present application, a training method for a TCM-assisted diagnosis large model is provided, comprising: obtaining a fine-tuning sample data set, wherein each fine-tuning sample data in the fine-tuning sample data set includes image data and marking information of the same user, the image data includes facial visible light images, tongue visible light images, and facial infrared images, and the marking information is used to describe symptom characteristic information of abnormal items in TCM theory corresponding to each image data in the fine-tuning sample data; using the fine-tuning sample data set to fine-tune a first multimodal large model to obtain a second multimodal large model for TCM-assisted diagnosis, wherein the first multimodal large model is a pre-trained multimodal large model with image processing capabilities, and the second multimodal large model is used to identify abnormal items using TCM theory based on the user's facial visible light images, tongue visible light images, and facial infrared images.

[0009] According to the above technical means, by constructing a fine-tuning sample data set containing the user's facial visible light image, tongue visible light image and facial infrared image, and combining the symptom characteristic information of abnormal items in TCM theory as a marker, the pre-trained multimodal large model is fine-tuned. This process effectively integrates multimodal image data with TCM theoretical knowledge, thereby generating a second multimodal large model specifically for TCM-assisted diagnosis. The model can accurately use TCM theory to identify abnormal items based on the user's facial, tongue and facial infrared images, significantly improving the intelligence level and accuracy of TCM recognition in online consultations.

[0010] Furthermore, obtaining a fine-tuning sample data set includes: obtaining original image data of multiple users, wherein at least one of the age, gender, and region of the multiple users is different; annotating the original image data corresponding to each user to obtain tag information corresponding to the original image data of each user; generating fine-tuning sample data corresponding to each user based on the original image data and tag information corresponding to each user; and combining the fine-tuning sample data corresponding to all users to obtain a fine-tuning sample data set.

[0011] According to the above technical means, the diversity and representativeness of the data set are ensured by widely collecting original image data of multiple users of different ages, genders, and geographical backgrounds. Furthermore, by carefully annotating these original image data, label information rich in TCM recognition information is generated, thereby constructing a high-quality fine-tuning sample data set. This process not only enriches the connotation of the data set, but also improves the usability of the data set. Finally, these fine-tuning sample data sets are combined to form a comprehensive and effective fine-tuning sample data set, which provides a solid foundation for the subsequent fine-tuning training of the TCM-assisted diagnosis multimodal large model. The implementation of this technical means has greatly improved the accuracy and efficiency of TCM recognition in online consultations.

[0012] Furthermore, the original image data corresponding to each user is annotated to obtain labeling information corresponding to the original image data of each user, including: annotating the original image data according to a feature description vocabulary to obtain labeling information, wherein the feature description vocabulary is a set of standard feature description vocabularies of TCM symptoms pre-constructed using TCM theory; and / or annotating the original image data according to the experience of TCM experts to obtain labeling information.

[0013] According to the above technical means, by comprehensively using the feature description vocabulary constructed by TCM theory and the experience and knowledge of TCM experts, the user's image data is accurately annotated, generating labeling information that not only meets TCM recognition standards but is also rich in expert experience. This method ensures the accuracy and authority of the annotation content, so that the generated fine-tuning sample data set can truly reflect the key features and symptoms in TCM recognition. Through this method, not only the quality of the data set is improved, but also more abundant and accurate learning materials are provided for subsequent multimodal large model training.

[0014] Furthermore, based on the original image data and marking information corresponding to each user, fine-tuning sample data corresponding to each user is generated, including: preprocessing the original image data corresponding to each user to obtain preprocessed first image data, wherein the preprocessing includes: data cleaning and data normalization; performing data enhancement on the first image data to obtain data enhanced second image data, wherein the data enhancement includes at least one of image rotation, image scaling and image cropping; combining the second image data and the marking information according to a target data structure to obtain target structure data; converting the target structure data into a target format to obtain fine-tuning sample data, wherein the target format is a format that can be processed by the first multimodal large model.

[0015] According to the above technical means, a series of refined image preprocessing and data enhancement steps are used to effectively improve the quality and diversity of fine-tuning sample data. Data cleaning and normalization ensure the accuracy and consistency of the data, laying the foundation for subsequent analysis; while data enhancement increases the complexity and representativeness of the samples through image rotation, scaling, cropping and other means, which helps the model learn more robust features. Finally, these processed image data and labeling information are combined into fine-tuning sample data in a specific structure and format, which not only facilitates the input and processing of the first multimodal large model, but also significantly improves the efficiency and effectiveness of model training. The implementation of this technical means provides high-quality sample data for the fine-tuning training of the TCM-assisted diagnosis multimodal large model.

[0016] Furthermore, the first multimodal large model is fine-tuned and trained using the fine-tuning sample data set to obtain a second multimodal large model for TCM-assisted diagnosis, including: dividing the fine-tuning sample data set into a test set and a training set; using the first fine-tuning sample data in the training set to fine-tune the first multimodal large model to obtain a second multimodal large model, wherein the fine-tuning training is iterative training using LoRA incremental learning.

[0017] According to the above technical means, by finely dividing the fine-tuning sample data set into a test set and a training set, and using the training set to perform fine-tuning training on the first multimodal large model based on LoRA incremental learning, a second multimodal large model focusing on TCM-assisted diagnosis was successfully obtained. This fine-tuning training method not only maintains the strong basic capabilities of the original model, but also optimizes it for the specific needs of TCM identification in online consultations, making the new model more accurate and efficient in identifying TCM symptom characteristics. In addition, the application of LoRA incremental learning makes the training process more flexible and efficient, reduces the consumption of computing resources, and accelerates the convergence speed of the model.

[0018] Furthermore, after obtaining the second multimodal large model for TCM-assisted diagnosis, it includes: inputting the second fine-tuning sample data in the test set into the second multimodal large model to obtain a target diagnosis result; comparing the target diagnosis result with the preset diagnosis result, and when the target diagnosis result is consistent with the preset diagnosis result, it is determined that the second multimodal large model has been verified, wherein the preset diagnosis result is obtained by a TCM expert through a pre-diagnosis of the image data in the second fine-tuning sample data based on TCM theory.

[0019] According to the above technical means, by inputting the sample data in the test set into the model and comparing its output results with the results of the pre-diagnosis of TCM experts based on TCM theory, the accuracy and reliability of the model can be fully verified. When the results of the two are consistent, it indicates that the second multimodal large model has been successfully verified and has the ability to be used in actual TCM-assisted diagnosis. The implementation of this technical means not only ensures the diagnostic accuracy of the model, but also enhances the practicality and credibility of the model.

[0020] Furthermore, after obtaining the second multimodal large model for TCM-assisted diagnosis, it also includes: using the second fine-tuned sample data in the test set to determine the accuracy of the second multimodal large model; evaluating the performance of the second multimodal large model based on the accuracy, and when the accuracy is greater than or equal to a preset accuracy threshold, determining that the second multimodal large model meets the needs of using TCM theory to identify and diagnose abnormal items in TCM syndromes in the user's image data.

[0021] According to the above technical means, evaluating its accuracy through test set data and judging whether the model performance meets the preset standards is a key step to ensure the actual application effect of the model. When the model accuracy is higher than or equal to the set threshold, it means that the model can accurately use TCM theory to identify and diagnose abnormal items in the user's image data, thereby effectively supporting TCM clinical decision-making. This technical means ensures the reliability and stability of the model through quantitative evaluation.

[0022] According to another aspect of the embodiment of the present application, a training device for a TCM-assisted diagnosis large model is also provided, including: a sample acquisition module, used to acquire a fine-tuning sample data set, wherein each fine-tuning sample data in the fine-tuning sample data set includes image data and marking information of the same user, the image data includes facial visible light images, tongue visible light images and facial infrared images, and the marking information is used to describe the symptom characteristic information of abnormal items in TCM theory corresponding to each image data in the fine-tuning sample data; a model training module, used to use the fine-tuning sample data set to fine-tune the first multimodal large model to obtain a second multimodal large model for TCM-assisted diagnosis, wherein the first multimodal large model is a pre-trained multimodal large model with image processing capabilities, and the second multimodal large model is used to identify abnormal items using TCM theory based on the user's facial visible light images, tongue visible light images and facial infrared images.

[0023] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein the memory is used to store computer programs; and the processor is used to execute the steps of the training method of the large model of TCM-assisted diagnosis in any of the above-mentioned embodiments by running the computer program stored in the memory.

[0024] According to another aspect of the embodiments of the present application, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to execute the training method steps of the TCM-assisted diagnosis large model in any of the above-mentioned embodiments when running.

[0025] Beneficial effects of this application:

[0026] This application constructs a fine-tuning sample data set containing visible light images of the user's face, visible light images of the tongue, and infrared images of the face, and combines the symptom characteristic information of abnormal items in traditional Chinese medicine theory as a marker to fine-tune the pre-trained multimodal large model. This process effectively integrates multimodal image data with theoretical knowledge of traditional Chinese medicine, thereby generating a second multimodal large model specifically for TCM-assisted diagnosis. The model can accurately use traditional Chinese medicine theory to identify abnormal items based on the user's face, tongue, and infrared facial images, significantly improving the intelligence level and accuracy of TCM identification in online consultations, thereby effectively solving the limitations of information transmission and uneven distribution of doctor resources in online TCM consultations, and improving the efficiency of online consultations and patients' medical experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0029] Figure 1 It is a schematic diagram of the hardware environment of an optional training method for a large model of TCM-assisted diagnosis provided in an embodiment of the present application;

[0030] Figure 2 It is a flowchart of an optional training method for a large model of TCM-assisted diagnosis provided in an embodiment of the present application;

[0031] Figure 3 It is a structural block diagram of an optional training device for a large model of TCM-assisted diagnosis provided in an embodiment of the present application;

[0032] Figure 4 It is a structural block diagram of an optional electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] According to one aspect of an embodiment of the present application, a training method for a TCM-assisted diagnosis large model is provided. Optionally, in this embodiment, the training method for the TCM-assisted diagnosis large model can be applied to a hardware environment consisting of a terminal and a server. The server is connected to the terminal via a network, and can be used to provide services for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server.

[0036] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal may not be limited to a PC, a mobile phone, a tablet computer, etc.

[0037] The training method of the TCM-assisted diagnosis large model in the embodiment of the present application can be executed by a server, or by a terminal, or by both a server and a terminal. The terminal can also execute the training method of the TCM-assisted diagnosis large model in the embodiment of the present application by a client installed thereon.

[0038] Taking the training method of the TCM-assisted diagnosis model in this embodiment executed by the server as an example, please refer to Figure 1 , Figure 1 is a schematic diagram of the hardware environment of an optional training method for a large model of TCM-assisted diagnosis provided in an embodiment of the present application, such as Figure 1As shown, the hardware environment of the training method of the TCM-assisted diagnosis large model includes: a terminal 102, and a server 104 connected to the terminal 102 through a network. The server 104 is used to deploy the first multimodal large model to be trained, which is a pre-trained multimodal large model with image recognition capabilities. The first multimodal large model executes the training method of the TCM-assisted diagnosis large model of the embodiment of the present application, and uses a fine-tuning sample data set for fine-tuning training; the terminal 102 is used to display the recognition result output by using the second multimodal large model after training to identify abnormal items of the user's facial visible light image, tongue visible light image, and facial infrared image according to TCM theoretical knowledge, wherein the recognition result includes facial abnormal items, tongue abnormal items, and facial temperature abnormal items. In specific practice, the recognition result also includes the TCM theoretical basis corresponding to each abnormal item.

[0039] The training method of the TCM-assisted diagnosis large model in this embodiment can be applied to scenarios such as online TCM consultation. Although online consultation has shown great potential in improving the accessibility of medical services, the current online consultation method based on text or voice description still has significant limitations. First of all, the lack of text description and unprofessionalism are one of the main factors restricting the accuracy of diagnosis. Patients often find it difficult to describe their symptoms in accurate and comprehensive TCM terms, which leads to deviations in the process of information transmission and even misleads the doctor's diagnostic thinking. This information asymmetry not only increases the risk of misdiagnosis, but also reduces the patient's medical experience and trust.

[0040] Secondly, although video calls make up for the shortcomings of text descriptions to a certain extent, by intuitively displaying information about the patient's face and other parts of the body, it provides doctors with a richer basis for diagnosis. However, this model also faces challenges: on the one hand, the scarcity of high-end medical resources makes it difficult for qualified Chinese medicine experts to take time out to participate in online diagnosis and treatment, making it difficult for online consultation platforms to attract and retain high-quality doctor resources; on the other hand, even if video is used, if the doctor is inexperienced and inexperienced, he may still make a one-sided diagnosis due to the inability to accurately identify the nature of the disease, affecting the treatment effect. It can be seen that the online Chinese medicine consultation service in the existing technology has limitations in information transmission and uneven distribution of doctor resources, resulting in low consultation efficiency.

[0041] To solve the above problems, this embodiment provides a training method for a large TCM-assisted diagnosis model running on the above server. Figure 2 , Figure 2 It is a flowchart of an optional training method of a large model of TCM-assisted diagnosis provided in an embodiment of the present application, such as Figure 2 As shown, the training method of the TCM-assisted diagnosis large model in the embodiment of the present application specifically includes the following steps:

[0042] Step S201, obtaining a fine-tuning sample data set, wherein each fine-tuning sample data in the fine-tuning sample data set includes image data and tag information of the same user, the image data includes a facial visible light image, a tongue visible light image, and a facial infrared image, and the tag information is used to describe symptom feature information of an abnormal item in TCM theory corresponding to each image data in the fine-tuning sample data;

[0043] Step S202, fine-tune the first multimodal large model using a fine-tuning sample data set to obtain a second multimodal large model for TCM-assisted diagnosis, wherein the first multimodal large model is a pre-trained multimodal large model with image processing capabilities, and the second multimodal large model is used to identify abnormal items based on the user's facial visible light image, tongue visible light image, and facial infrared image using TCM theory.

[0044] Through the above steps S201 to S202, by constructing a fine-tuning sample data set including the visible light image of the user's face, the visible light image of the tongue and the infrared image of the face, and combining the symptom characteristic information of the abnormal items in the theory of traditional Chinese medicine as a marker, the pre-trained multimodal large model is fine-tuned and trained. This process effectively integrates multimodal image data with theoretical knowledge of traditional Chinese medicine, thereby generating a second multimodal large model specifically for TCM-assisted diagnosis. The model can accurately use the theory of traditional Chinese medicine to identify abnormal items based on the user's face, tongue and infrared images of the face, significantly improving the intelligence level and accuracy of TCM identification in online consultations, and solving the problem of low consultation efficiency caused by the limitations of information transmission and uneven distribution of doctor resources in online TCM consultation services in related technologies.

[0045] Combine the following Figure 2 The training method of the TCM-assisted diagnosis large model in the embodiment of the present application is explained.

[0046] In the technical solution of step S201, a fine-tuning sample data set is obtained. The fine-tuning sample data set is used to fine-tune the pre-trained first multimodal large model. In this embodiment, the fine-tuning sample data set includes a plurality of fine-tuning sample data, each of which includes image data and marking information of the same user, the image data includes a facial visible light image, a tongue visible light image, and a facial infrared image, and the marking information is used to describe the symptom characteristic information of the abnormal item in the theory of traditional Chinese medicine corresponding to each image data in the fine-tuning sample data.

[0047] It should be noted that, in this embodiment, the first multimodal large model is a pre-trained multimodal large model with image processing capabilities. An optional specific implementation is that the first multimodal large model adopts the PaLiGemma model, which can receive image and text inputs, generate text outputs, and support a variety of visual and language tasks. Users can fine-tune the model according to specific tasks to obtain better performance. At the same time, the PaLiGemma model also supports transformers integration, which is convenient for users to carry out secondary development and deployment. The PaliGemma model consists of SigLIP-So400m as an image encoder and Gemma-2B as a text decoder. SigLIP is an advanced model that can parse images and texts at the same time. It works similarly to CLIP, including joint training of image and text encoders. Gemma is a decoder model designed specifically for text generation. The image encoding function of SigLIP is combined with Gemma through a linear adapter to make the PaLiGemma model a powerful visual language model.

[0048] As an optional embodiment, obtaining a fine-tuning sample data set includes: obtaining original image data of multiple users, wherein at least one of the age, gender, and region of the multiple users is different; annotating the original image data corresponding to each user to obtain tag information corresponding to the original image data of each user; generating fine-tuning sample data corresponding to each user based on the original image data and tag information corresponding to each user; and combining the fine-tuning sample data corresponding to all users to obtain a fine-tuning sample data set.

[0049] In this embodiment, in order to ensure the diversity of samples, the original image data of multiple users are obtained, wherein at least one of the age, gender and region of the multiple users is different. The original image data of each user includes the visible light image of the user's face, the visible light image of the tongue and the infrared image of the face. The visible light image of the face, the visible light image of the tongue and the infrared image of the face of the user can be directly collected from the high-definition photos of the face and tongue of multiple users and the corresponding infrared images of the face, or the high-definition photos of the face and tongue of multiple users and the corresponding infrared images of the face can be directly taken using an image acquisition device. The image acquisition device includes a camera, a high-definition camera, etc. for collecting high-definition photos of the face and tongue, and also includes an infrared camera or a thermal imager, etc. for collecting infrared images of the face.

[0050] In actual practice, when the image acquisition device is used to collect visible light images of the face, visible light images of the tongue, and infrared images of the face, a standardized acquisition process is adopted to make the collected images easier to analyze and process. This process is similar to taking a photo of an ID card. The dotted line position of the face is marked in the acquisition system. The photo is considered successful only when the face is within the dotted line and the distance is appropriate. Otherwise, a voice prompt is given to remind the user to make adjustments. The tongue photo is similar. The tip of the tongue and the lip position are marked. The photo is considered successful only when it is within this range.

[0051] It is understandable that by widely collecting raw image data from multiple users of different ages, genders, and geographical backgrounds, the diversity and representativeness of the data set are ensured. Furthermore, by carefully annotating these raw image data, label information rich in TCM recognition information is generated, thereby constructing a high-quality fine-tuning sample data set. This process not only enriches the connotation of the data set, but also improves the usability of the data set. Finally, these fine-tuning sample data sets are combined to form a comprehensive and effective fine-tuning sample data set, which provides a solid foundation for the subsequent fine-tuning training of the TCM-assisted diagnosis multimodal large model. The implementation of this technical means has greatly improved the accuracy and efficiency of TCM recognition in online consultations.

[0052] As an optional embodiment, the original image data corresponding to each user is annotated to obtain labeling information corresponding to the original image data of each user, including: annotating the original image data according to a feature description vocabulary to obtain labeling information, wherein the feature description vocabulary is a set of standard feature description vocabularies of TCM symptoms pre-constructed using TCM theory; and / or annotating the original image data according to the experience of TCM experts to obtain labeling information.

[0053] In this embodiment, the original image data can be annotated using a feature description vocabulary, or a Chinese medicine expert can directly annotate the original image data based on experience, or a combination of the two can be used for annotation, with each image corresponding to annotated one tag information. The feature description vocabulary is pre-constructed, and when it is constructed, a Chinese medicine expert will first annotate the tag information corresponding to a part of the Chinese medicine symptom characteristics based on experience, and then normalize it based on the expert's tag information and the theoretical knowledge of Chinese medicine, so as to establish a set of standard feature description vocabulary that combines Chinese medicine experience and Chinese medicine theory. In order to make the annotated information more accurate, the feature description vocabulary can be updated according to the theoretical knowledge of Chinese medicine, such as when a new symptom appears, the descriptive word corresponding to the symptom is updated to the feature description vocabulary. Before a large number of samples are annotated according to the standard feature vocabulary in the future, the purpose of formulating the vocabulary first is to ensure that the feature descriptions of all samples are consistent, thereby promoting the accuracy and efficiency of the annotation, so that the subsequent model training can be based on unified and clear data features.

[0054] It is understandable that by comprehensively using the feature description vocabulary constructed by TCM theory and the experience and knowledge of TCM experts, the user's image data is accurately annotated, generating labeling information that not only meets TCM recognition standards but is also rich in expert experience. This method ensures the accuracy and authority of the annotation content, so that the generated fine-tuning sample data set can truly reflect the key features and symptoms in TCM recognition. Through this method, not only the quality of the data set is improved, but also more abundant and accurate learning materials are provided for subsequent multimodal large model training.

[0055] As an optional embodiment, fine-tuning sample data corresponding to each user is generated according to the original image data and marking information corresponding to each user, including: preprocessing the original image data corresponding to each user to obtain preprocessed first image data, wherein the preprocessing includes: data cleaning and data normalization; performing data enhancement on the first image data to obtain data enhanced second image data, wherein the data enhancement includes at least one of image rotation, image scaling and image cropping; combining the second image data and the marking information according to a target data structure to obtain target structure data; converting the target structure data into a target format to obtain fine-tuning sample data, wherein the target format is a format that can be processed by the first multimodal large model.

[0056] When generating fine-tuning sample data corresponding to each user based on the original image data and marking information corresponding to each user, the original image needs to be preprocessed. The preprocessing may include data cleaning and data normalization. The specific steps of each preprocessing operation are not specifically limited in this embodiment. It is also necessary to perform data enhancement on the preprocessed first image data. The data enhancement includes at least one of image rotation, image scaling, and image cropping. The specific steps of each data enhancement operation are not specifically limited in this embodiment. Finally, the data-enhanced second image data and marking information are combined according to the target data structure, and the combined target structure data is format-converted according to the target format to obtain fine-tuning sample data, wherein the target format is a format that can be processed by the first multimodal large model.

[0057] In specific practice, the data-enhanced second image data and marking information are combined according to the target data structure, which actually converts the annotated marking information into structured data, namely, target structure data. The data structures of the facial visible light image, tongue visible light image, facial infrared image and corresponding marking information in the converted second image data are the same, and are all target data structures. In this embodiment, the target data structure can be {"image":"the address corresponding to each picture", "text":"the marking information corresponding to each picture"}. Taking the tongue visible light image and marking information as an example, the target structure data obtained after conversion according to the target data structure is as follows:

[0058] {"image":"Address of tongue image","text":"The tongue is light red with a purplish tint and the tongue coating is thin and white. These signs indicate that the patient may have liver depression and spleen deficiency. The tongue image shows slight stagnation of qi and blood."}.

[0059] In specific practice, different multimodal large language models may have different format requirements for input data, so the target structure data needs to be formatted according to the target format so that it can meet the requirements of the first multimodal large model. In this embodiment, the first multimodal large model is the PaLiGemma model, and the format of the input data of the PaLiGemma model is mainly JSONL (JSON Lines) format. The JSONL format is a lightweight data exchange format based on JSON. It stores each JSON object as a line of text, so that data can be easily read and written line by line. Therefore, in this embodiment, the target format is JSONL format.

[0060] It is understandable that through a series of refined image preprocessing and data enhancement steps, the quality and diversity of the fine-tuning sample data are effectively improved. Data cleaning and normalization ensure the accuracy and consistency of the data, laying the foundation for subsequent analysis; while data enhancement increases the complexity and representativeness of the samples through image rotation, scaling, cropping and other means, which helps the model learn more robust features. Finally, these processed image data and labeling information are combined into fine-tuning sample data in a specific structure and format, which not only facilitates the input and processing of the first multimodal large model, but also significantly improves the efficiency and effectiveness of model training. The implementation of this technical means provides high-quality sample data for the fine-tuning training of the TCM-assisted diagnosis multimodal large model.

[0061] In the technical solution of step S202, the first multimodal large model is fine-tuned and trained using a fine-tuning sample data set to obtain a second multimodal large model for TCM-assisted diagnosis, wherein the first multimodal large model is a pre-trained multimodal large model with image processing capabilities, and the second multimodal large model is used to identify abnormal items based on the user's facial visible light image, tongue visible light image, and facial infrared image using TCM theory.

[0062] As an optional embodiment, a fine-tuning sample data set is used to fine-tune the first multimodal large model to obtain a second multimodal large model for TCM-assisted diagnosis, including: dividing the fine-tuning sample data set into a test set and a training set; using the first fine-tuning sample data in the training set to fine-tune the first multimodal large model to obtain a second multimodal large model, wherein the fine-tuning training is iterative training using LoRA incremental learning.

[0063] In this embodiment, the main function of the training set is to train the first multimodal large model. The training process is an iterative process. The first multimodal large model will traverse the training set many times, and after each iteration, the weight will be adjusted according to the difference between the predicted result and the true label until a certain stop condition is reached, such as the number of iterations, the performance is no longer improved, or a certain performance threshold is reached. The main function of the test set is to evaluate the performance of the second multimodal large model after training. After the model training is completed, we need an independent data set to test the generalization ability of the second multimodal large model after training to new and unseen data. This independent data set is the test set. The test set should be independent of the training set. Through the test set, we can obtain the performance indicators of the second multimodal large model in real scenarios, such as accuracy, recall rate, F1 score, etc. In specific practice, the test set and the training set can be randomly split, and can be in proportion, such as 7:3, or 8:2, and the validation set may not be available.

[0064] It should be noted that LoRA (Low-Rank Adaptation) is a lightweight fine-tuning technology. Its core idea is to use fewer training parameters to approximate the incremental parameters obtained by full parameter fine-tuning, so as to achieve efficient fine-tuning with less memory usage. LoRA assumes that when the model adapts to a specific task, the learned incremental parameter matrix (ΔW) is low-rank, that is, there are redundant parameters or highly correlated parameter matrices, while the actual effective parameters are of lower dimensions. Therefore, LoRA approximates ΔW by low-rank decomposition, thereby reducing the number of parameters.

[0065] In this embodiment, LoRA incremental learning specifically includes the following steps:

[0066] 1. Freeze the pre-trained model weights: Before starting LoRA fine-tuning, first freeze all the weights of the pre-trained first multimodal model, that is, these weights will not change during the fine-tuning process.

[0067] 2. Insert trainable low-rank decomposition matrices: In each layer of the pre-trained first multimodal model, insert trainable low-rank decomposition matrices A and B (where the dimension of A is much smaller than the dimension of the original weight matrix). These matrices are used to approximate the incremental parameter matrix ΔW.

[0068] 3. Training low-rank decomposition matrices: During the training process, only the inserted low-rank decomposition matrices A and B are trained to learn the knowledge of a specific task (i.e., identifying abnormal items using theoretical knowledge of traditional Chinese medicine).

[0069] 4. Merge weights for inference: In the inference phase, the trained low-rank decomposition matrices A and B are merged into the weights of the original pre-trained first multimodal model to obtain the model weights optimized for a specific task. Since the number of parameters is greatly reduced, the training process is more efficient and consumes less computing resources, and the merging operation is linear, so there is no additional overhead during inference.

[0070] It is understandable that by finely dividing the fine-tuning sample data set into a test set and a training set, and using the training set to fine-tune the first multimodal large model based on LoRA incremental learning, a second multimodal large model focusing on TCM-assisted diagnosis was successfully obtained. This fine-tuning training method not only maintains the strong basic capabilities of the original model, but also optimizes it for the specific needs of TCM identification in online consultations, making the new model more accurate and efficient in identifying TCM symptom characteristics. In addition, the application of LoRA incremental learning makes the training process more flexible and efficient, reduces the consumption of computing resources, and accelerates the convergence of the model.

[0071] As an optional embodiment, after obtaining the second multimodal large model for TCM-assisted diagnosis, it includes: inputting the second fine-tuning sample data in the test set into the second multimodal large model to obtain a target diagnosis result; comparing the target diagnosis result with the preset diagnosis result, and when the target diagnosis result is consistent with the preset diagnosis result, it is determined that the second multimodal large model has been verified, wherein the preset diagnosis result is obtained by a TCM expert through a pre-diagnosis of the image data in the second fine-tuning sample data based on TCM theory.

[0072] After the pre-trained first multimodal large model is trained, the output results of the second multimodal large model need to be reviewed to determine whether the model can process the image. In specific practice, the target diagnosis result output by the second multimodal large model is compared with the preset diagnosis result. When the target diagnosis result is consistent with the preset diagnosis result, it is determined that the second multimodal large model has passed the verification. The preset diagnosis result is obtained by a Chinese medicine expert who pre-diagnoses the image data in the second fine-tuning sample data based on Chinese medicine theory.

[0073] It is understandable that by inputting the sample data in the test set into the model and comparing its output results with the results of the pre-diagnosis of TCM experts based on TCM theory, the accuracy and reliability of the model can be fully verified. When the results of the two are consistent, it indicates that the second multimodal large model has been successfully verified and has the ability to be used in actual TCM-assisted diagnosis. The implementation of this technical means not only ensures the diagnostic accuracy of the model, but also enhances the practicality and credibility of the model.

[0074] As an optional embodiment, after obtaining the second multimodal large model for TCM-assisted diagnosis, it also includes: using the second fine-tuning sample data in the test set to determine the accuracy of the second multimodal large model; evaluating the performance of the second multimodal large model based on the accuracy, and when the accuracy is greater than or equal to a preset accuracy threshold, it is determined that the second multimodal large model meets the needs of using TCM theory to identify and diagnose abnormal items in TCM syndromes in the user's image data.

[0075] The accuracy is one of the important indicators for evaluating the performance of a classification model, which indicates the proportion of samples correctly predicted by the model to the total number of samples. In this embodiment, the specific steps for determining the accuracy of the second multimodal large model using the second fine-tuning sample data in the test set are no longer specifically limited.

[0076] It is understandable that evaluating the accuracy of the test set data and judging whether the model performance meets the preset standards is a key step to ensure the actual application effect of the model. When the model accuracy is higher than or equal to the set threshold, it means that the model can accurately use TCM theory to identify and diagnose abnormal items in the user's image data, thereby effectively supporting TCM clinical decision-making. This technical means ensures the reliability and stability of the model through quantitative evaluation.

[0077] In actual applications, the user's facial visible light photo and facial diagnosis prompt are first input into the fine-tuned second multimodal large model, which outputs facial abnormalities and related TCM conclusions; secondly, the user's tongue visible light photo and tongue diagnosis prompt are input into the fine-tuned second multimodal large model, which outputs tongue abnormalities and related TCM conclusions; finally, the user's facial infrared image and infrared thermal imaging prompt are input into the fine-tuned second multimodal large model, which outputs facial temperature abnormalities and related TCM conclusions.

[0078] It can be understood that the second multimodal large model can not only accurately use TCM theory to identify abnormal items based on the user's face, tongue and facial infrared images, significantly improving the intelligence level and accuracy of TCM recognition in online consultations, but also effectively avoid misdiagnosis due to human negligence, greatly improving the accuracy of diagnosis. At the same time, by defining a standardized feature description vocabulary, it not only reduces the subjective differences between doctors, but also promotes the standardization of TCM diagnosis, and introduces quantifiable and reproducible scientific evaluation standards for TCM face-to-face consultations, thereby effectively solving the limitations of information transmission and uneven distribution of doctor resources in online TCM consultations, and improving the accuracy of online consultations and patients' medical experience.

[0079] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0080] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0081] According to another aspect of the embodiment of the present application, a training device for a TCM-assisted diagnosis large model for implementing the training method of the TCM-assisted diagnosis large model is also provided. Figure 3 , Figure 3 is a structural block diagram of an optional training device for a large model of TCM-assisted diagnosis provided in an embodiment of the present application, such as Figure 3 As shown, the device may include:

[0082] The sample acquisition module 301 is used to acquire a fine-tuning sample data set, wherein each fine-tuning sample data in the fine-tuning sample data set includes image data and tag information of the same user, the image data includes a facial visible light image, a tongue visible light image, and a facial infrared image, and the tag information is used to describe the symptom feature information of the abnormal item in the theory of traditional Chinese medicine corresponding to each image data in the fine-tuning sample data;

[0083] The model training module 302 is used to use the fine-tuning sample data set to fine-tune the first multimodal large model to obtain a second multimodal large model for TCM-assisted diagnosis, wherein the first multimodal large model is a pre-trained multimodal large model with image processing capabilities, and the second multimodal large model is used to identify abnormal items based on the user's facial visible light image, tongue visible light image and facial infrared image using TCM theory.

[0084] It should be noted that the sample acquisition module 301 in this embodiment can be used to execute the above step S201, and the model training module 302 in this embodiment can be used to execute the above step S202.

[0085] Regarding the training device of the TCM-assisted diagnosis big model in this embodiment, the specific manner in which its sample acquisition module 301 and model training module 302 execute the above-mentioned TCM-assisted diagnosis big model training method has been described in detail in the embodiment of the TCM-assisted diagnosis big model training method, and will not be elaborated here.

[0086] It can be understood that the technical solution provided in this embodiment, each module in the training device of the TCM-assisted diagnosis large model, constructs a fine-tuning sample data set including the user's facial visible light image, tongue visible light image and facial infrared image, and combines the symptom characteristic information of abnormal items in TCM theory as a marker to fine-tune the pre-trained multimodal large model. This process effectively integrates multimodal image data with TCM theoretical knowledge, thereby generating a second multimodal large model specifically for TCM-assisted diagnosis. The model can accurately use TCM theory to identify abnormal items based on the user's facial, tongue and facial infrared images, significantly improving the intelligence level and accuracy of TCM identification in online consultations, thereby effectively solving the limitations of information transmission and uneven distribution of doctor resources in online TCM consultations, and improving the accuracy of online consultations and patients' medical experience.

[0087] The device in this embodiment, in addition to the above-mentioned modules, may also include a module for executing any method in the embodiments of the training method of any of the aforementioned TCM-assisted diagnosis large models.

[0088] It should be noted that the above modules and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can be run in the implementation of Figure 1 The hardware environment of the method shown may be implemented by software or hardware, wherein the hardware environment includes a network environment.

[0089] According to another aspect of the embodiment of the present application, an electronic device for implementing the training method of the above-mentioned TCM-assisted diagnosis large model is also provided. The electronic device can be a server, a terminal, or a combination thereof.

[0090] According to another embodiment of the present application, an electronic device is also provided. Figure 4 , Figure 4 is a structural block diagram of an optional electronic device provided in an embodiment of the present application, such as Figure 4 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0091] Memory 1503, used for storing computer programs;

[0092] The processor 1501 is used to implement the following steps when executing the program stored in the memory 1503:

[0093] Step S201, obtaining a fine-tuning sample data set, wherein each fine-tuning sample data in the fine-tuning sample data set includes image data and tag information of the same user, the image data includes a facial visible light image, a tongue visible light image, and a facial infrared image, and the tag information is used to describe symptom feature information of an abnormal item in TCM theory corresponding to each image data in the fine-tuning sample data;

[0094] Step S202, fine-tune the first multimodal large model using a fine-tuning sample data set to obtain a second multimodal large model for TCM-assisted diagnosis, wherein the first multimodal large model is a pre-trained multimodal large model with image processing capabilities, and the second multimodal large model is used to identify abnormal items based on the user's facial visible light image, tongue visible light image, and facial infrared image using TCM theory.

[0095] It can be understood that the technical solution provided in this embodiment is that the processor of the electronic device constructs a fine-tuning sample data set including the visible light image of the user's face, the visible light image of the tongue and the infrared image of the face, and combines the symptom characteristic information of the abnormal items in the theory of traditional Chinese medicine as a marker to fine-tune the pre-trained multimodal large model. This process effectively integrates the multimodal image data with the theoretical knowledge of traditional Chinese medicine, thereby generating a second multimodal large model specifically for TCM-assisted diagnosis. The model can accurately use the theory of traditional Chinese medicine to identify abnormal items based on the user's face, tongue and infrared images of the face, significantly improving the intelligence level and accuracy of TCM identification in online consultations, thereby effectively solving the limitations of information transmission and the uneven distribution of doctor resources in online TCM consultations, and improving the accuracy of online consultations and patients' medical experience.

[0096] Optionally, in this embodiment, the above-mentioned communication bus can be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above-mentioned electronic device and other devices.

[0097] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0098] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0099] An embodiment of the present application further provides a computer-readable storage medium, the storage medium including a stored program, wherein the method steps of the above method embodiment are executed when the program is run.

[0100] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0101] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0102] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to execute all or part of the steps of the methods of each embodiment of the present application.

[0103] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0104] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0105] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0106] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0107] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A training method for a large model of TCM-assisted diagnosis, characterized in that: include: Acquire a fine-tuning sample data set, wherein each fine-tuning sample data in the fine-tuning sample data set includes image data and tag information of the same user, the image data includes a facial visible light image, a tongue visible light image, and a facial infrared image, and the tag information is used to describe symptom feature information of an abnormal item in traditional Chinese medicine theory corresponding to each image data in the fine-tuning sample data; The first multimodal large model is fine-tuned and trained using the fine-tuning sample data set to obtain a second multimodal large model for TCM-assisted diagnosis, wherein the first multimodal large model is a pre-trained multimodal large model with image processing capabilities, and the second multimodal large model is used to identify abnormal items based on the user's facial visible light image, tongue visible light image, and facial infrared image using TCM theory.

2. The training method of the TCM-assisted diagnosis large model according to claim 1 is characterized in that: The step of obtaining a fine-tuning sample data set includes: Acquire original image data of a plurality of users, wherein at least one of age, gender, and region of the plurality of users is different; Labeling the original image data corresponding to each user to obtain label information corresponding to the original image data of each user; Generate fine-tuning sample data corresponding to each user based on the original image data and label information corresponding to each user; The fine-tuning sample data sets corresponding to all users are gathered together to obtain the fine-tuning sample data set.

3. The training method of the TCM-assisted diagnosis large model according to claim 2 is characterized in that: The labeling of the original image data corresponding to each user to obtain the label information corresponding to the original image data of each user includes: The original image data is annotated according to a feature description vocabulary to obtain the tag information, wherein the feature description vocabulary is a set of standard feature description vocabulary of TCM symptoms pre-constructed using TCM theory; and / or, According to the experience of TCM experts, the original image data is annotated to obtain the marking information.

4. The training method of the TCM-assisted diagnosis large model according to claim 2 is characterized in that: Generating fine-tuning sample data corresponding to each user according to the original image data and the marking information corresponding to each user includes: Preprocessing the original image data corresponding to each user to obtain preprocessed first image data, wherein the preprocessing includes: data cleaning and data normalization; Performing data enhancement on the first image data to obtain data-enhanced second image data, wherein the data enhancement includes at least one of image rotation, image scaling, and image cropping; Combining the second image data and the marking information according to a target data structure to obtain target structure data; The target structure data is format-converted according to a target format to obtain the fine-tuning sample data, wherein the target format is a format that can be processed by the first multimodal large model.

5. The training method of the TCM-assisted diagnosis large model according to claim 1 is characterized in that: The method of using the fine-tuning sample data set to fine-tune the first multimodal large model to obtain a second multimodal large model for TCM-assisted diagnosis includes: Dividing the fine-tuning sample data set into a test set and a training set; Using the first fine-tuning sample data in the training set, fine-tune the first multimodal large model to obtain the second multimodal large model, wherein the fine-tuning training is iterative training using LoRA incremental learning.

6. The training method of the TCM-assisted diagnosis large model according to claim 5 is characterized in that: After obtaining the second multimodal large model for TCM auxiliary diagnosis, including: Inputting the second fine-tuning sample data in the test set into the second multimodal large model to obtain a target diagnosis result; The target diagnosis result is compared with the preset diagnosis result. When the target diagnosis result is consistent with the preset diagnosis result, it is determined that the second multimodal large model verification has passed, wherein the preset diagnosis result is obtained by a Chinese medicine expert through a pre-diagnosis of the image data in the second fine-tuning sample data based on Chinese medicine theory.

7. The training method of the TCM-assisted diagnosis large model according to claim 5 is characterized in that: After obtaining the second multimodal large model for TCM-assisted diagnosis, it also includes: Determine the accuracy of the second multimodal large model using the second fine-tuning sample data in the test set; The performance of the second multimodal large model is evaluated according to the accuracy rate. When the accuracy rate is greater than or equal to a preset accuracy rate threshold, it is determined that the second multimodal large model meets the requirements of identifying and diagnosing abnormal items in TCM syndromes of the user's image data using TCM theory.

8. A training device for a large model of TCM-assisted diagnosis, characterized in that: include: A sample acquisition module is used to acquire a fine-tuning sample data set, wherein each fine-tuning sample data in the fine-tuning sample data set includes image data and tag information of the same user, the image data includes a facial visible light image, a tongue visible light image, and a facial infrared image, and the tag information is used to describe the symptom feature information of the abnormal item in the theory of traditional Chinese medicine corresponding to each image data in the fine-tuning sample data; The model training module is used to use the fine-tuning sample data set to fine-tune the first multimodal large model to obtain a second multimodal large model for TCM-assisted diagnosis, wherein the first multimodal large model is a pre-trained multimodal large model with image processing capabilities, and the second multimodal large model is used to identify abnormal items based on the user's facial visible light image, tongue visible light image and facial infrared image using TCM theory.

9. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface and the memory communicate with each other via the communication bus, wherein: The memory is used to store computer programs; The processor is used to execute the training method of the TCM-assisted diagnosis large model as described in any one of claims 1 to 7 by running the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the training method of the TCM-assisted diagnosis large model as described in any one of claims 1 to 7 when running.

Images