Multi-modal model traditional Chinese medicine tongue diagnosis method and system
Through the multimodal model, the tongue image image and diagnostic information are processed, and the intuitive disease type and probability values are output, which solves the problem that the tongue diagnosis results of traditional Chinese medicine in the prior art are not intuitive enough, and improves the accuracy and comprehensibility of the diagnosis.
Patent Information
- Application Number
- CN202411084715.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The diagnostic results obtained by the existing digital technology of tongue diagnosis in traditional Chinese medicine are not intuitive enough, relying on professional knowledge and experience, and are difficult to understand and interpret.
Using the traditional Chinese medicine tongue diagnosis method based on multimodal model, by obtaining tongue image images and preliminary diagnostic information, image encoding and text encoding are performed, and the characteristics are fused to input the multimodal model, and the intuitive disease type and probability value are output to reduce the Reliance on physician experience.
It improves the intuitiveness and accuracy of diagnostic results, allowing patients to quickly understand their own illness and reduce their dependence on doctors.
Smart Images

Figure CN118942638A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of auxiliary medical technology, and in particular to a traditional Chinese medicine tongue diagnosis method and system based on a multimodal model. Background Art
[0002] Tongue diagnosis is an important part of TCM diagnosis. Physicians usually observe the color, shape, and coating of the tongue to help judge the health status of the human body. However, traditional tongue diagnosis often relies on the professional knowledge and experience of TCM physicians, and has high requirements for their diagnostic experience.
[0003] With the advancement of modern science and technology, more and more digital technologies are being introduced into TCM tongue diagnosis. These digital technologies can assist TCM physicians in their diagnosis to a certain extent and reduce their reliance on their experience. However, most of the diagnostic results obtained by the current digital technology of TCM tongue diagnosis are not intuitive enough and still have a strong degree of professionalism. They often require professional TCM physicians to correctly interpret and diagnose based on their knowledge and experience. Summary of the invention
[0004] In view of this, the present invention provides a TCM tongue diagnosis method and system based on a multimodal model to solve the problem that the diagnosis results obtained according to the TCM tongue diagnosis digitization technology in the prior art are not intuitive enough.
[0005] In a first aspect, the present invention provides a method for tongue diagnosis in traditional Chinese medicine based on a multimodal model, the method comprising:
[0006] Obtain the user's tongue image to be diagnosed and the doctor's preliminary diagnosis information of the user;
[0007] Input the tongue image to be diagnosed and the preliminary diagnosis information into the pre-established multimodal tongue diagnosis model, and output the model diagnosis results, which include: tongue image characteristics and predicted pathology;
[0008] Based on the model diagnosis results, predict the user's disease type and the disease probability value corresponding to the disease type.
[0009] The model diagnosis results and diagnosis prediction results given by the TCM tongue diagnosis method based on the multimodal model in this embodiment are more easy to understand, so that patients can understand their medical conditions intuitively and quickly, reducing their dependence on doctors.
[0010] In an optional implementation, the tongue image to be diagnosed includes a reference object, and after acquiring the tongue image to be diagnosed, the following steps are performed:
[0011] Identify the positions of the tongue image and the reference object in the tongue image to be diagnosed, wherein the color of the reference object is white;
[0012] Segment the tongue image and reference object;
[0013] The color of the tongue image is corrected using a reference object to obtain a corrected tongue image to be diagnosed, and the corrected tongue image to be diagnosed is used to input into a multimodal tongue diagnosis model.
[0014] In this embodiment, by performing color correction on the tongue image through a reference object, the influence of ambient light on the image color can be effectively reduced, thereby making the collected image as standardized as possible and improving the accuracy of the model diagnosis results and diagnostic prediction results output by the model.
[0015] In an optional implementation, after obtaining the preliminary diagnosis information, the following steps are included:
[0016] The preliminary diagnosis information is subjected to text information preprocessing to obtain preprocessed preliminary diagnosis information, and the preprocessed preliminary diagnosis information is used to input into a multimodal tongue diagnosis model.
[0017] Inputting the preprocessed text into the tongue image multimodal model can effectively improve the accuracy of the output results of the tongue image multimodal model.
[0018] In an optional implementation, after the tongue image to be diagnosed and the preliminary diagnosis information are input into the pre-established multimodal tongue diagnosis model, the following steps are included:
[0019] Performing image coding on the tongue image to be diagnosed to obtain tongue image coding information;
[0020] Text coding is performed on the preliminary diagnosis information to obtain the text coding information of the tongue image;
[0021] Perform feature fusion on tongue image coding information and tongue text coding information;
[0022] The fused features are input into the core computing unit in the multimodal tongue diagnosis model for multimodal information processing to obtain the multimodal information processing result, which is then output as the model diagnosis result.
[0023] In an optional implementation, based on the model diagnosis result, predicting the user's disease type and the disease probability value corresponding to the disease type includes:
[0024] The model diagnosis results are input into the pre-trained text information diagnosis model to obtain the disease type and the disease probability value corresponding to the disease type. The text information diagnosis model is trained by the health diagnosis results of healthy tongue images and the disease diagnosis results of diseased tongue images. In this way, ordinary patients can see their own diagnosis results very intuitively.
[0025] In an optional embodiment, the multimodal tongue diagnosis model is established by the following steps:
[0026] Acquire a number of tongue image samples after color correction and diagnostic information samples corresponding to the tongue image samples, wherein the tongue image samples include healthy tongue image samples and diseased tongue image samples, and the diagnostic information samples corresponding to the diseased tongue image samples include tongue image features and pathological features;
[0027] Extracting image features from tongue image samples to obtain a tongue image feature set corresponding to the tongue image samples;
[0028] Extracting semantic features from the diagnostic information samples to obtain a diagnostic information feature set corresponding to the diagnostic information samples;
[0029] Fusing the tongue image feature set and the diagnostic information feature set to obtain a fused feature set;
[0030] The fused feature set is input into the initial multimodal model architecture and the model is trained. After the training is completed, a multimodal tongue diagnosis model is obtained.
[0031] During the construction of the multimodal tongue diagnosis model, a large number of data sets containing tongue images and detailed medical history information were collected to train and verify the performance of the initial multimodal model in TCM tongue diagnosis identification and disease diagnosis, thereby effectively improving diagnostic efficiency and accuracy.
[0032] In a second aspect, the present invention provides a traditional Chinese medicine tongue diagnosis system based on a multimodal model, the system comprising:
[0033] An acquisition module, used to acquire the user's tongue image to be diagnosed and the doctor's preliminary diagnosis information of the user;
[0034] The model prediction module is used to input the tongue image to be diagnosed and the preliminary diagnosis information into the pre-established multimodal tongue diagnosis model, and output the model diagnosis results, which include: tongue image characteristics and predicted pathology;
[0035] The probability prediction module is used to predict the user's disease type and the disease probability value corresponding to the disease type based on the model diagnosis results.
[0036] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multimodal model-based Chinese medicine tongue diagnosis method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0037] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions for enabling a computer to execute the TCM tongue diagnosis method based on a multimodal model according to the first aspect or any corresponding embodiment thereof.
[0038] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the TCM tongue diagnosis method based on a multimodal model according to the first aspect or any corresponding embodiment thereof.
[0039] It should be noted that the TCM tongue diagnosis system based on a multimodal model, the computer device, and the computer-readable storage medium provided by the present invention correspond to the TCM tongue diagnosis method based on a multimodal model. Therefore, for the beneficial effects of the TCM tongue diagnosis system based on a multimodal model, the computer device, and the computer-readable storage medium, please refer to the description of the corresponding beneficial effects of the TCM tongue diagnosis method based on a multimodal model above, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0041] Figure 1 is a flow chart of a method for traditional Chinese medicine tongue diagnosis based on a multimodal model according to an embodiment of the present invention;
[0042] Figure 2 is a schematic diagram of a reference object according to an embodiment of the present invention;
[0043] Figure 3 is a schematic diagram of an application example of TCM tongue diagnosis based on a multimodal model according to an embodiment of the present invention;
[0044] Figure 4 is another schematic diagram of a process of TCM tongue diagnosis based on a multimodal model according to an embodiment of the present invention;
[0045] Figure 5 is a schematic diagram of multiple application examples of TCM tongue diagnosis based on a multimodal model according to an embodiment of the present invention;
[0046] Figure 6 is a structural block diagram of a traditional Chinese medicine tongue diagnosis system based on a multimodal model according to an embodiment of the present invention;
[0047] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0049] Most of the diagnostic results obtained by the digital technology of TCM tongue diagnosis only provide data on the color, shape, and wetness of the tongue coating. These data are not intuitive enough for people without professional knowledge and experience, and TCM physicians still need to combine their clinical experience and TCM theory to give the final diagnosis and treatment recommendations.
[0050] In view of this, according to an embodiment of the present invention, an embodiment of a traditional Chinese medicine tongue diagnosis method based on a multimodal model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.
[0051] In this embodiment, a TCM tongue diagnosis method based on a multimodal model is provided, which can be executed by a server, a terminal, a mobile terminal and other devices. Figure 1 is a flow chart of a TCM tongue diagnosis method based on a multimodal model according to an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:
[0052] Step S101, obtaining the user's tongue image to be diagnosed and the doctor's preliminary diagnosis information of the user.
[0053] The tongue image to be diagnosed can be taken with a high-precision camera or mobile phone. Since the shooting environment and lighting conditions of each patient are different, in order to minimize the image color difference caused by factors such as ambient light. Figure 2 As shown, a standard A4 white paper can be used as a reference object to facilitate the subsequent correction of the color difference of the tongue image. The user in this embodiment can be a healthy patient or a patient who is under observation or actually suffers from a disease.
[0054] At the same time, in order to make full use of the image and text processing capabilities of the multimodal model, when collecting tongue image data, some physicians' preliminary diagnosis information on the patient can also be recorded, such as non-image data such as basic patient information, symptom description, and physical fitness assessment. These non-image data can be saved in the form of text as preliminary diagnosis information. Among them, a patient can generate a preliminary diagnosis information, and one or more tongue images to be diagnosed can be collected.
[0055] Step S102, input the tongue image to be diagnosed and the preliminary diagnosis information into a pre-established multimodal tongue diagnosis model, and output the model diagnosis result, which includes: tongue image characteristics and predicted pathology.
[0056] Reference Figure 3 As shown in the figure, the tongue image to be diagnosed is used as the system input, and after preprocessing, it is input into the multimodal tongue diagnosis model. After the learning and analysis of the multimodal tongue diagnosis model, the model diagnosis result is output. The output model diagnosis result is as follows:
[0057] "This picture shows a patient's tongue. From the perspective of traditional Chinese medicine, it has the following characteristics:
[0058] 1. Thick and greasy tongue coating: The tongue is covered with a thick layer of white moss-like substance, which is usually considered to be a manifestation of dampness, phlegm or spleen and stomach dysfunction.
[0059] 2. Red tongue: The tongue itself is red in color, which may indicate the presence of heat evil or yin deficiency and fire excess in the body.
[0060] 3. Full tongue: The tongue looks fuller, without obvious cracks or bruises, which may mean that the Qi and blood are more abundant.
[0061] Take thick and greasy tongue coating as an example, where the tongue image characteristic is: "The tongue surface is covered with a thick layer of white fur-like substance" and the predicted pathological manifestation is: "This is usually considered to be a manifestation of dampness, phlegm or spleen and stomach dysfunction". The model diagnosis result output by the multimodal tongue diagnosis model in this embodiment not only specifically reflects the true state of the tongue image, but also gives the possible pathology of the patient, so that the patient can understand the current physical condition in a simple and intuitive way.
[0062] Step S103: predicting the user's disease type and the disease probability value corresponding to the disease type based on the model diagnosis result.
[0063] After the model diagnosis result is output according to the multimodal tongue diagnosis model, the patient's disease type and disease probability can be further predicted based on the model diagnosis result and the pre-established text information diagnosis model. Specifically, a variety of possible disease types can be predicted according to the model diagnosis result, and the disease probability value can be further predicted for a specific disease type. If the choice is to predict a variety of possible disease types according to the model diagnosis result, the disease probability value corresponding to each disease type can be predicted, and the disease types above a certain probability value, such as 0.5 points and the corresponding disease probability value are output. If the choice is to predict a certain disease type according to the model diagnosis result, the disease type and the corresponding disease probability value can be directly output. In this embodiment, predicting a variety of possible disease types according to the model diagnosis result is more suitable for predicting the model diagnosis result for one patient, predicting a certain disease type according to the model diagnosis result, and more suitable for predicting the model diagnosis result for multiple patients. For a disease type, it has certain selectivity.
[0064] Reference Figure 3 As shown, according to the model diagnosis result, the diagnosis result prediction is further performed, and it is predicted that the patient has a high probability of being infected with Helicobacter pylori (HP), and the probability value of the disease is 0.82. In this embodiment, the higher the probability value of the disease, the higher the probability of the disease. The diagnosis prediction result allows the patient to intuitively understand his or her own diagnosis result, further reducing the dependence on the physician's experience.
[0065] At present, traditional Chinese medicine tongue diagnosis is highly subjective and has a low degree of standardization, and the diagnostic results predicted based on digital technology are difficult to understand. The traditional Chinese medicine tongue diagnosis method based on a multimodal model proposed in this embodiment uses deep learning technology to construct a new tongue diagnosis multimodal model. The model integrates computer vision and natural language processing technology to achieve synchronous analysis and deep fusion of tongue diagnosis image features and accompanying text descriptions of the disease, effectively improving the objectivity and accuracy of tongue diagnosis results, and showing excellent generalization ability and clinical application value in a variety of disease diagnosis scenarios, providing new ideas and technical support for promoting the modernization and standardization of traditional Chinese medicine tongue diagnosis and the development of intelligent medicine.
[0066] Furthermore, the model diagnosis results and diagnosis prediction results given by the TCM tongue diagnosis method based on a multimodal model in this embodiment are more easy to understand, allowing patients to intuitively and quickly understand their own medical conditions and reduce their dependence on doctors.
[0067] In some optional implementations, the tongue image to be diagnosed includes a reference object, and after acquiring the tongue image to be diagnosed, the following steps are performed:
[0068] Step S201, identifying the positions of the tongue image and the reference object in the tongue image to be diagnosed, wherein the color of the reference object is white, specifically, the reference object can be white paper.
[0069] Step S202, segment the tongue image and the reference object. In this embodiment, it is necessary to distinguish the tongue location and the reference object location, and the corresponding image area can be extracted using the object segmentation and recognition algorithm. Figure 3 As shown, the tongue and paper areas are segmented.
[0070] Step S203, using the reference object to correct the color of the tongue image to obtain a corrected tongue image to be diagnosed, and the corrected tongue image to be diagnosed is used to input the multimodal tongue diagnosis model.
[0071] The color correction algorithm of the tongue image can be called. In this algorithm, the color of the A4 white paper area is used as the standard white (i.e. 255, 255, 255), and the colors of other areas are corrected accordingly according to this reference. Through this correction algorithm, the influence of ambient light on the image color can be further reduced, so that the collected image can be standardized as much as possible. Figure 3 Figure 2. Color correction of tongue image shows the tongue image after color correction.
[0072] In this embodiment, by performing color correction on the tongue image through a reference object, the influence of ambient light on the image color can be effectively reduced, thereby making the collected image as standardized as possible and improving the accuracy of the model diagnosis results and diagnostic prediction results output by the model.
[0073] In some optional implementations, after obtaining the preliminary diagnosis information, the following steps are included:
[0074] Step S204, the preliminary diagnosis information is subjected to text information preprocessing to obtain preprocessed preliminary diagnosis information, and the preprocessed preliminary diagnosis information is used to input into the multimodal tongue diagnosis model.
[0075] In addition to color correction of tongue images, refer to Figure 4 As shown, in this embodiment, the physician's preliminary diagnosis information is also simply preprocessed, and the preprocessing may include text cleaning, text segmentation, keyword extraction, keyword standardization, etc. The preprocessed text and color-corrected tongue image are input into the tongue image multimodal model, which can effectively improve the accuracy of the output results of the tongue image multimodal model.
[0076] Reference Figure 4 As shown, in some optional implementations, after the tongue image to be diagnosed and the preliminary diagnosis information are input into the pre-established multimodal tongue diagnosis model, the following steps are included:
[0077] Step S301, performing image coding on the tongue image to be diagnosed to obtain tongue image coding information.
[0078] Step S302, text encoding the preliminary diagnosis information to obtain tongue image text encoding information.
[0079] Step S303, feature fusion of tongue image coding information and tongue text coding information.
[0080] Step S304, input the fused features into the core computing unit in the multimodal tongue diagnosis model, that is, the multimodal tongue diagnosis model body, to perform multimodal information processing, obtain the multimodal information processing result, and use the multimodal information processing result as the model diagnosis result for output.
[0081] The preprocessed text and color-corrected tongue image will be used as the input of the multimodal tongue diagnosis model. The tongue image will be encoded and the text information will be encoded. These encoded information will enter the core computing unit of the multimodal tongue diagnosis model for unified calculation. Although the multimodal tongue diagnosis model has a certain generalization ability, in order to further improve the algorithm effect, the multimodal large model will be fine-tuned using annotated data. The model fine-tuned using annotated data can further understand the relationship between different modal data, such as the correspondence between tongue features and symptoms, thereby further improving the accuracy of the diagnostic output.
[0082] In some optional implementations, based on the model diagnosis result, predicting the user's disease type and the disease probability value corresponding to the disease type includes:
[0083] The model diagnosis results are input into a pre-trained text information diagnosis model, the disease type and the disease probability value corresponding to the disease type. The text information diagnosis model is trained by the health diagnosis results of healthy tongue images and the disease diagnosis results of diseased tongue images.
[0084] The output of the multimodal tongue diagnosis model is a feature description of the tongue image and some related diagnoses, but these diagnoses are generally only understood by professional Chinese medicine practitioners. In order to further reduce the threshold for the use of the algorithm, the algorithm can output very intuitive diagnostic results. In this embodiment, a text information diagnosis and judgment algorithm is adopted. The algorithm is essentially a text classifier, which can be trained using the diagnostic description of normal people and the diagnostic description of patients. The final result of the text classifier is a score, that is, the probability of illness in this embodiment. The score is within the range of 0-1. The smaller the score, the lower the probability of suffering from a certain disease. On the contrary, the larger the score, the higher the probability of suffering from a certain disease. Through this design, ordinary patients can see their own diagnostic results very intuitively.
[0085] Reference Figure 5As shown, the final input and output results of multiple tongue images are displayed. From the results, it can be seen that the method has a certain accuracy in the application of using tongue images and preliminary diagnostic information to predict Helicobacter pylori infection.
[0086] In some optional embodiments, the multimodal tongue diagnosis model is established by the following steps:
[0087] Step S401, obtaining a number of tongue image samples after color correction and diagnostic information samples corresponding to the tongue image samples, wherein the tongue image samples include healthy tongue image samples and diseased tongue image samples, and the diagnostic information samples corresponding to the diseased tongue image samples include tongue image characteristics, pathological characteristics, disease type, etc.
[0088] Step S402: extract image features from the tongue image samples to obtain a tongue image feature set that corresponds one-to-one to the tongue image samples.
[0089] Step S403: extracting semantic features from the diagnosis information samples to obtain a diagnosis information feature set corresponding to the diagnosis information samples.
[0090] Step S404, fusing the tongue image feature set and the diagnostic information feature set to obtain a fused feature set.
[0091] Step S405, input the fused feature set into the initial multimodal model architecture and perform model training. After the training is completed, a multimodal tongue diagnosis model is obtained.
[0092] During the construction of the multimodal tongue diagnosis model, a large number of data sets containing tongue images and detailed medical history information were collected to train and verify the performance of the initial multimodal model in TCM tongue diagnosis identification and disease diagnosis, thereby effectively improving diagnostic efficiency and accuracy.
[0093] Furthermore, the TCM tongue diagnosis method based on the multimodal model provided in this embodiment has broad application prospects, which are mainly reflected in the following aspects:
[0094] Medical diagnosis assistance system: This method can be directly applied to medical institutions to assist TCM physicians in conducting more objective and accurate tongue diagnosis, thereby improving diagnostic efficiency and accuracy. This is particularly important for primary medical institutions and telemedicine scenarios, as it can make up for the lack of professional TCM resources and allow more patients to enjoy high-quality TCM services.
[0095] Health management platform: Combining mobile Internet and wearable devices, develop health management software for individual users. Users can take and upload tongue images to obtain instant health status analysis and suggestions, promote personal health management and disease prevention, and open up the health consultation and preventive medical market.
[0096] Intelligent TCM education: As a teaching tool, it helps TCM students and practitioners better understand the relationship between tongue images and diseases, standardize the learning of tongue diagnosis skills, and accelerate the modernization of TCM education.
[0097] Scientific research cooperation and product development: Cooperate with pharmaceutical companies and scientific research institutions to use the model for early screening of new drug development, disease mechanism research, etc., or develop targeted health products and treatment plans to expand scientific research and product applications.
[0098] Insurance and health management: Insurance companies can use this technology to assess customer health risks and develop personalized insurance products. At the same time, they can provide health management services to reduce compensation risks and increase customer stickiness.
[0099] Personalized medical solutions: Combine the patient's tongue diagnosis results with other medical data to customize personalized treatment plans, promote the development of precision medicine, and meet the needs of the high-end medical market.
[0100] International promotion: As a representative achievement of the modernization of traditional Chinese medicine, this technology helps to internationalize traditional Chinese medicine, attract overseas users and medical institutions to adopt it, and expand the influence and acceptance of traditional Chinese medicine around the world.
[0101] In summary, this technology can not only improve the scientificity and convenience of TCM diagnosis and treatment, but also promote the deep integration of TCM and modern technology, open up new market space, and bring significant economic benefits and social value. With the continuous optimization of technology and the expansion of application scenarios, its potential will be further released.
[0102] In the present embodiment, a Chinese medicine tongue diagnosis system based on a multimodal model is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.
[0103] This embodiment provides a traditional Chinese medicine tongue diagnosis system based on a multimodal model. Figure 6 As shown, the system includes:
[0104] The acquisition module 501 is used to acquire the tongue image to be diagnosed of the user and the preliminary diagnosis information of the user by the physician;
[0105] The model prediction module 502 is used to input the tongue image to be diagnosed and the preliminary diagnosis information into the pre-established multimodal tongue diagnosis model, and output the model diagnosis result, which includes: tongue image characteristics and predicted pathology;
[0106] The probability prediction module 503 is used to predict the user's disease type and the disease probability value corresponding to the disease type based on the model diagnosis result.
[0107] In some optional embodiments, the system further comprises:
[0108] The color correction module is used to identify the positions of the tongue image and the reference object in the tongue image to be diagnosed, wherein the color of the reference object is white; segment the tongue image and the reference object; use the reference object to correct the color of the tongue image to obtain a corrected tongue image to be diagnosed, and the corrected tongue image to be diagnosed is used to input the multimodal tongue diagnosis model.
[0109] The text preprocessing module is used to preprocess the text information of the preliminary diagnosis information to obtain the preprocessed preliminary diagnosis information, and the preprocessed preliminary diagnosis information is used to input the multimodal tongue diagnosis model.
[0110] In some optional implementations, the model prediction module 502 includes:
[0111] The model prediction unit is used to perform image encoding on the tongue image to be diagnosed to obtain tongue image encoding information; perform text encoding on the preliminary diagnosis information to obtain tongue image text encoding information; perform feature fusion on the tongue image encoding information and the tongue image text encoding information; input the fused features into the core operation unit in the multimodal tongue diagnosis model to perform multimodal information processing to obtain the multimodal information processing result, and use the multimodal information processing result as the model diagnosis result for output.
[0112] In some optional implementations, the probability prediction module 503 includes:
[0113] The probability prediction unit is used to input the model diagnosis results into a pre-trained text information diagnosis model to obtain the disease type and the disease probability value corresponding to the disease type. The text information diagnosis model is trained by the health diagnosis results of healthy tongue images and the disease diagnosis results of diseased tongue images.
[0114] In some optional embodiments, the system further comprises:
[0115] A multimodal tongue diagnosis model construction module is used to obtain a number of tongue image samples after color correction and diagnostic information samples corresponding to the tongue image samples, wherein the tongue image samples include healthy tongue image samples and diseased tongue image samples, and the diagnostic information samples corresponding to the diseased tongue image samples include tongue features and pathological features; image feature extraction is performed on the tongue image samples to obtain a tongue image feature set corresponding to the tongue image samples one by one; semantic feature extraction is performed on the diagnostic information samples to obtain a diagnostic information feature set corresponding to the diagnostic information samples one by one; feature fusion is performed on the tongue image feature set and the diagnostic information feature set to obtain a fused feature set; the fused feature set is input into the initial multimodal model architecture and the model is trained. After the training is completed, a multimodal tongue diagnosis model is obtained.
[0116] The TCM tongue diagnosis system based on the multimodal model in this embodiment is presented in the form of functional units, where the units refer to ASIC circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above functions.
[0117] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0118] The embodiment of the present invention also provides a computer device having the above Figure 6 The Traditional Chinese Medicine tongue diagnosis system based on a multimodal model is shown.
[0119] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.
[0120] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0121] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0122] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0123] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0124] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.
[0125] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0126] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0127] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A TCM tongue diagnosis method based on a multimodal model, It is characterized in that The method comprises: Obtaining the user's tongue image to be diagnosed and the physician's preliminary diagnosis information of the user; Inputting the tongue image to be diagnosed and the preliminary diagnosis information into a pre-established multimodal tongue diagnosis model, and outputting a model diagnosis result, wherein the model diagnosis result includes: tongue image characteristics and predicted pathology; Based on the model diagnosis result, the user's disease type and the disease probability value corresponding to the disease type are predicted.
2. The method according to claim 1, It is characterized in that The tongue image to be diagnosed includes a reference object. After acquiring the tongue image to be diagnosed, the method includes: Identifying the tongue image and the position of the reference object in the tongue image to be diagnosed, wherein the color of the reference object is white; Segmenting the tongue image and the reference object; The color of the tongue image is corrected using the reference object to obtain a corrected tongue image to be diagnosed, and the corrected tongue image to be diagnosed is used to input the multimodal tongue diagnosis model.
3. The method according to claim 1 or 2, It is characterized in that After obtaining the preliminary diagnostic information, the following steps are performed: The preliminary diagnosis information is subjected to text information preprocessing to obtain preprocessed preliminary diagnosis information, and the preprocessed preliminary diagnosis information is used to input into the multimodal tongue diagnosis model.
4. The method according to claim 1, It is characterized in that After the tongue image to be diagnosed and the preliminary diagnosis information are input into the pre-established multimodal tongue diagnosis model, the method includes: Performing image coding on the tongue image to be diagnosed to obtain tongue image coding information; Performing text encoding on the preliminary diagnosis information to obtain tongue image text encoding information; Performing feature fusion on the tongue image coding information and the tongue image text coding information; The fused features are input into the core computing unit in the multimodal tongue diagnosis model to perform multimodal information processing to obtain a multimodal information processing result, which is then output as the model diagnosis result.
5. The method according to claim 1, It is characterized in that The predicting of the user's disease type and the disease probability value corresponding to the disease type based on the model diagnosis result includes: The model diagnosis result is input into a pre-trained text information diagnosis model to obtain the disease type and the disease probability value corresponding to the disease type. The text information diagnosis model is trained by the health diagnosis results of healthy tongue images and the disease diagnosis results of diseased tongue images.
6. The method according to claim 1, It is characterized in that The multimodal tongue diagnosis model is established by the following steps: Acquire a plurality of tongue image samples after color correction and diagnostic information samples corresponding to the tongue image samples, wherein the tongue image samples include healthy tongue image samples and diseased tongue image samples, and the diagnostic information samples corresponding to the diseased tongue image samples include tongue image features and pathological features; Extracting image features from the tongue image samples to obtain a tongue image feature set corresponding to the tongue image samples one by one; Extracting semantic features from the diagnostic information samples to obtain diagnostic information feature sets that correspond one-to-one to the diagnostic information samples; Performing feature fusion on the tongue image feature set and the diagnostic information feature set to obtain a fused feature set; The fused feature set is input into the initial multimodal model architecture and model training is performed. After the training is completed, the multimodal tongue diagnosis model is obtained.
7. A Chinese medicine tongue diagnosis system based on multimodal model, It is characterized in that The system comprises: An acquisition module, used to acquire the tongue image of the user to be diagnosed and the preliminary diagnosis information of the user by the physician; A model prediction module, used for inputting the tongue image to be diagnosed and the preliminary diagnosis information into a pre-established multimodal tongue diagnosis model, and outputting a model diagnosis result, wherein the model diagnosis result includes: tongue image characteristics and predicted pathology; The probability prediction module is used to predict the user's disease type and the disease probability value corresponding to the disease type based on the model diagnosis result.
8. A computer device, It is characterized in that include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multimodal model-based TCM tongue diagnosis method according to any one of claims 1 to 6 by executing the computer instructions.
9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the traditional Chinese medicine tongue diagnosis method based on a multimodal model as described in any one of claims 1-6.
10. A computer program product, It is characterized in that It comprises computer instructions, and the computer instructions are used to enable a computer to execute the traditional Chinese medicine tongue diagnosis method based on a multimodal model as described in any one of claims 1-6.
Citation Information
Patent Citations
Intelligent processing method for tongue image information of stomach disease in traditional Chinese medicine based on deep learning
CN108986912A
Deep learning-based skin disease tongue picture classification model, establishment method and application
CN115410046A
Tongue diagnosis multi-modal data classification method and system and storage medium
CN116313040A
Prescription recommendation method and system based on tongue image and knowledge graph
CN116844687A
Tongue picture recognition method and device based on multi-modal information and electronic equipment
CN117611581A
Cited By
Traditional Chinese medicine tongue diagnosis and prescription recommendation system based on multi-modal feature fusion
CN120089345A
Tongue picture disease risk prediction method and system based on prompt mutual learning
CN121171582A
A tongue disease risk prediction method and system based on prompt mutual learning
CN121171582B
