Method and device for carrying out tongue picture analysis by using AI large model
Through AI big model analysis of tongue objects, the problem of poor accuracy and robustness of traditional tongue diagnosis methods is solved, and high accuracy and comprehensive analysis of intelligent tongue diagnosis on mobile phones is achieved, providing more accurate health assessment and suggestion.
Patent Information
- Application Number
- CN202411073565.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The traditional tongue diagnosis method relies on experienced Chinese medicine practitioners, and has problems with low recognition accuracy and poor robustness, so it cannot be promoted to smart tongue diagnosis applications such as mobile phone shooting.
The AI model is used to analyze tongue images, and reorganize the results of tongue images in different dimensions by obtaining tongue images recognition results, determine the system prompts, and use the AI model to analyze tongue images in an asynchronous and concurrent manner, including physical constitution analysis, symptom analysis and comprehensive interpretation with suggestions.
It improves the accuracy and comprehensiveness of tongue image analysis, can realize intelligent tongue diagnosis on mobile phones, and provides more accurate and scientific health assessment and suggestions.
Smart Images

Figure CN119069137A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and device for tongue image analysis using an AI large model. Background Art
[0002] Tongue diagnosis is an important means of diagnosis in traditional Chinese medicine. By observing the tongue (including tongue spirit, tongue color, tongue shape, coating, coating color, etc.), the health status of the human body can be reflected. However, the traditional tongue diagnosis method relies on experienced Chinese medicine practitioners, and its diagnosis results have certain subjectivity and uncertainty. The existing automated tongue image recognition methods mainly use simple classification algorithms or feature matching methods, which have problems such as low recognition accuracy and poor robustness. Therefore, they are generally used in special tongue diagnosis devices and cannot be extended to mobile phone photography. Therefore, there is an urgent need for a more accurate and comprehensive tongue image recognition method that allows intelligent tongue diagnosis through a mobile phone.
[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0004] The embodiments of the present invention provide a method and device for tongue image analysis using an AI large model, so as to at least solve the technical problem of inaccurate tongue image analysis.
[0005] According to one aspect of an embodiment of the present invention, a method for tongue image analysis using an AI big model is provided, comprising: obtaining tongue image recognition results of different dimensions, and reorganizing the tongue image recognition results of different dimensions to obtain corresponding natural language descriptions; determining system prompts of the AI big model for different subtasks, wherein the different subtasks include a constitution analysis subtask, a symptom analysis subtask, and a comprehensive interpretation and suggestion subtask;
[0006] Based on the system prompts of each of the different subtasks and the corresponding natural language descriptions, the AI big model is used in an asynchronous and concurrent manner to perform tongue image analysis to obtain tongue image analysis results.
[0007] According to another aspect of an embodiment of the present invention, there is also provided a device for performing tongue image analysis using an AI big model, comprising: a reorganization module, configured to obtain tongue image recognition results of different dimensions, and reorganize the tongue image recognition results of different dimensions to obtain corresponding natural language descriptions; a determination module, configured to determine the system prompts of the AI big model for different subtasks, wherein the different subtasks include a constitution analysis subtask, a symptom analysis subtask, and a comprehensive interpretation and suggestion subtask; an analysis module, configured to perform tongue image analysis using the AI big model in an asynchronous and concurrent manner based on the system prompts of each of the different subtasks and the corresponding natural language descriptions to obtain tongue image analysis results.
[0008] In an embodiment of the present invention, tongue image recognition results of different dimensions are obtained, and the tongue image recognition results of different dimensions are reorganized to obtain corresponding natural language descriptions; system prompts of the AI big model are determined for different subtasks, wherein the different subtasks include constitution analysis subtasks, symptom analysis subtasks, and comprehensive interpretation and suggestion subtasks; based on the system prompts of each of the different subtasks and the corresponding natural language descriptions, the AI big model is used to perform tongue image analysis in an asynchronous and concurrent manner to obtain tongue image analysis results, thereby solving the technical problem of inaccurate tongue image analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0010] Figure 1 is a flow chart of an intelligent tongue diagnosis and analysis method according to an embodiment of the present invention;
[0011] Figure 2 is a flow chart of a method for tongue image analysis using an AI big model according to an embodiment of the present invention;
[0012] Figure 3 is a flow chart of a method for scoring health based on tongue diagnosis analysis results according to an embodiment of the present invention;
[0013] Figure 4 is a flow chart of another intelligent tongue diagnosis and analysis method according to an embodiment of the present invention;
[0014] Figure 5 is a tongue image captured using a bbox with margins after segmentation according to an embodiment of the present invention;
[0015] Figure 6 is a tongue image diagram in which a fissure area is delineated using yolov8 according to an embodiment of the present invention;
[0016] Figure 7 is a tongue image diagram in which the acupuncture area is delineated by using yolov8 according to an embodiment of the present invention;
[0017] Figure 8 is a tongue image diagram in which a tooth mark area is delineated by using yolov8 according to an embodiment of the present invention;
[0018] Fig. 9 is a classification diagram of system prompts for intelligent tongue diagnosis according to an embodiment of the present invention;
[0019] Fig.10is a flow chart of another method for tongue image analysis using an AI big model according to an embodiment of the present invention;
[0020] Fig.11 (a), (b), (c) and (d) are interface diagrams of the smart tongue diagnosis APP according to an embodiment of the present invention;
[0021] Fig.12 is a flow chart of another method for scoring health based on tongue diagnosis analysis results according to an embodiment of the present invention;
[0022] Fig.13 is a structural schematic diagram of an intelligent tongue diagnosis and analysis device according to an embodiment of the present invention;
[0023] Fig.14 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0026] Example 1
[0027] According to an embodiment of the present invention, an embodiment of an intelligent tongue diagnosis and analysis method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0028] The present application embodiment provides an intelligent tongue diagnosis and analysis method, such as Figure 1 As shown, the method comprises the following steps:
[0029] Step S102, obtaining a tongue image of the target object, and performing segmentation processing on the tongue image using a segmentation model to separate the tongue area of the target object from the background area to obtain a tongue area image, wherein the tongue image includes the tongue area and the background area.
[0030] Step S104, using the classification model to perform tongue image classification of different dimensions on the tongue area image to obtain tongue image classification results of different dimensions, and using the instance segmentation model to perform local tongue image instance segmentation and analysis on the tongue area image to obtain local tongue image analysis results.
[0031] Firstly, a classification model is used to perform tongue image classification of different dimensions on the tongue region image to obtain tongue image classification results of different dimensions.
[0032] The classification model is used to classify the tongue image in different dimensions for the tongue region image, wherein the different dimensions include tongue quality dimension, tongue color dimension, tongue shape-old and tender dimension, tongue shape-fat and thin dimension, coating-thickness dimension, coating-dryness dimension, coating-root dimension, coating-greasy and rotten dimension, and coating color dimension;
[0033] Using the classification model, for each different dimension, the tongue area image under the dimension is classified into different sub-dimensions of tongue image, and the tongue image classification results of the different dimensions are obtained, wherein the tongue spirit dimension includes a healthy tongue dimension and a dry tongue dimension; the tongue color dimension includes a light red tongue dimension, a light white tongue dimension, a red tongue dimension, a crimson tongue dimension, and a cyanotic tongue dimension; the tongue shape-old and tender dimension includes an old tongue dimension and a tender tongue dimension; the tongue shape-fat and thin dimension includes a fat tongue dimension and a thin tongue dimension; the moss-thickness dimension includes a thin moss dimension and a thick moss dimension; the moss-moist and dry dimension includes a moist moss dimension and a dry moss dimension; the moss-with or without roots dimension includes a rooted moss dimension and a rootless moss dimension; the moss-greasy, rotten and peeled dimension includes a greasy moss dimension, a rotten moss dimension and a peeled moss dimension, and the moss color dimension includes a white moss dimension, a yellow moss dimension and a gray-black moss dimension.
[0034] Secondly, the instance segmentation model is used to perform local tongue image instance segmentation and analysis on the tongue region image to obtain the analysis result of the local tongue image.
[0035] For example, the example segmentation model is used to identify and segment the crack region, the puncture region, and the tooth mark region in the tongue region image; the crack region, the puncture region, and the tooth mark region are analyzed respectively to obtain the analysis result of the local tongue image. For example, the areas of the crack region, the puncture region, and the tooth mark region are calculated respectively; the areas of the crack region, the puncture region, and the tooth mark region are compared with the corresponding preset area thresholds respectively, and the analysis result of the local tongue image is obtained based on the comparison result.
[0036] Step S106, generating a tongue image analysis result of the target object based on the tongue image classification results of different dimensions and the analysis results of the local tongue image.
[0037] The tongue image classification results of different dimensions and the analysis results of the local tongue image are integrated to obtain tongue image recognition results of different dimensions, and the tongue image recognition results of different dimensions are reorganized to obtain corresponding natural language descriptions; based on the natural language description, the AI big model is used to generate the tongue image analysis results. For example, the system prompts of the AI big model are determined for different subtasks, wherein the different subtasks include constitution analysis subtasks, symptom analysis subtasks, and comprehensive interpretation and suggestion subtasks; based on the system prompts of each of the different subtasks and the corresponding natural language descriptions, the AI big model is used to perform tongue image analysis in an asynchronous and concurrent manner to obtain the tongue image analysis results.
[0038] In some embodiments, the method of using the AI big model to analyze tongue image is as follows Figure 2 As shown, the following steps are included:
[0039] Step S202, obtaining tongue image recognition results of different dimensions, and reorganizing the tongue image recognition results of different dimensions to obtain corresponding natural language descriptions;
[0040] The tongue image recognition results of different dimensions in the form of computer code are parsed to obtain the natural language description corresponding to the computer code.
[0041] Step S204, determining prompts of the AI big model for different subtasks, wherein the different subtasks include a constitution analysis subtask, a symptom analysis subtask, and a comprehensive interpretation and suggestion subtask;
[0042] The prompts are determined for the different subtasks based on preset specifications, wherein the preset specifications include the identity of the analyst, the analyst's analysis and reasoning ideas, and the output format.
[0043] Step S206, based on the prompts of each of the different subtasks and the corresponding natural language description, the AI big model is used in an asynchronous and concurrent manner to perform tongue image analysis to obtain a tongue image analysis result.
[0044] First, the first AI big model in the AI big model is used to perform a physical analysis on the physical analysis subtask to generate an analysis result of the physical analysis dimension; the second AI big model in the AI big model is used to perform a symptom analysis on the symptom analysis subtask to generate an analysis result of the symptom analysis dimension; the third AI big model in the AI big model is used to perform a comprehensive interpretation and suggestion sub-analysis on the comprehensive interpretation and suggestion subtask to generate an analysis result of the comprehensive dimension;
[0045] Then, based on the analysis results of the constitution analysis dimension, the analysis results of the symptom analysis dimension, and the analysis results of the comprehensive dimension, the tongue image analysis result is generated. For example, the analysis results of the constitution analysis dimension, the analysis results of the symptom analysis dimension, and the analysis results of the comprehensive dimension are subjected to data format verification and correction; the verified and corrected analysis results of the constitution analysis dimension, the analysis results of the symptom analysis dimension, and the analysis results of the comprehensive dimension are merged to obtain the tongue image analysis result.
[0046] In some embodiments, after obtaining the analysis results of different dimensions, the health of the target object can also be scored, such as Figure 3 As shown, the scoring method includes the following steps:
[0047] Step S302, obtaining tongue image recognition results of different dimensions of the target object, and determining a plurality of tongue image items detected based on the tongue image recognition results;
[0048] Step S304, dividing the plurality of tongue image items into tongue image items with fixed deduction points and tongue image items with deduction points based on area ratio according to preset rules;
[0049] The tongue image items that are deducted based on the area ratio include: fissure tongue, puncture tongue, and tooth mark tongue. The tongue image items that are deducted with fixed points include at least one of the following tongue image items: healthy tongue, dry tongue, pale white tongue, pale red tongue, red tongue, crimson tongue, cyanotic tongue, old tongue, tender tongue, fat tongue, thin tongue, thin fur, thick fur, moist fur, dry fur, greasy fur, rotten fur, peeled fur, fur with roots, fur without roots, white fur, yellow fur, and gray-black fur.
[0050] Step S306, for the tongue image items with fixed deductions, determine the fixed deduction value based on the preset deduction value, and for the tongue image items with area ratio deductions, calculate the area deduction value based on the corresponding tongue image area ratio; based on the fixed deduction value and the area deduction value, generate the total health score of the target object.
[0051] Calculate the area ratio of the tongue image corresponding to the tongue image item based on the area ratio deduction; based on the area ratio, use the graded deduction method to obtain the area deduction value. For example, obtain multiple preset ratio threshold intervals; determine the ratio threshold interval in which the area ratio falls, and map the area ratio to the score of the gear corresponding to the ratio threshold interval in which it falls, so as to determine the area deduction value.
[0052] Example 2
[0053] Many existing tongue image analysis methods are traditional classification or feature matching methods with low accuracy and poor robustness. This application adopts a method combining small models with large models for tongue image recognition, detection and comprehensive analysis. The small model mainly uses a YOLO model for instance segmentation of the tongue, and then uses a CNN classification model such as ResNet to classify the tongue image's tongue spirit, tongue color, tongue shape, coating, and coating color, and uses YOLO instance segmentation to detect tooth-marked tongue, puncture tongue, and fissure tongue. Through the combination of this series of small models, accurate multi-dimensional recognition of tongue images is achieved, which is more accurate than some existing methods.
[0054] In addition, the tongue analysis of this application adopts the LLM large model. Compared with the traditional method based on fixed rules, this application inputs the recognition results into the language large model for reasoning and analysis, which is more comprehensive and can provide more specific analysis and suggestions. In the past, the effectiveness of the rule-based method depended on the rationality and meticulousness of the rules; by combining the small model with the large model, the method of this application not only improves the accuracy of recognition, but also enhances the comprehensiveness and rationality of the analysis.
[0055] The present application embodiment provides another intelligent tongue diagnosis and analysis method, such as Figure 4 As shown, the method comprises the following steps:
[0056] Step S402, input an image.
[0057] The user takes or uploads an image of the tongue.
[0058] Step S404, YOLO instance segmentation tongue area:
[0059] Use the YOLO (You Only Look Once) model to perform instance segmentation on the input tongue image and separate the tongue area from the background.
[0060] Different people take tongue images with different positions; and the system wants users to be able to perform tongue diagnosis with their mobile phones; therefore, it must adapt to different users' photos. If these images are directly pushed to subsequent processing, there will be a lot of interference, resulting in inaccurate recognition.
[0061] The existing tongue segmentation method mainly uses the color similarity of the color channel to segment. Since the color of the tongue is different from that of the facial skin, color segmentation can be used. However, this method is limited to typical tongue colors, and the selection of color intervals is very critical. For example, if the red color interval is selected, it may not perform well on a tongue with thick tongue coating, and mis-segmentation of the tongue and lips is also inevitable. Some other methods that use the watershed algorithm to segment the tongue area also have the problem of low robustness, especially for the root of the tongue, and when the light is uneven, the segmentation is not accurate and easy to misjudge.
[0062] This application uses a segmentation model, such as the instance segmentation model YoloV8 in deep learning, to perform instance segmentation on the tongue; first, an instance segmentation annotation dataset of the tongue is created, and then an instance segmentation model based on yolov8 is trained; then the trained customized model is used for reasoning to obtain the segmentation of the tongue, and the segmentation will output two data objects, polygon and bbox.
[0063] However, if we use the segmentation based on the edge, it is difficult to mark the subsequent processing such as the fatness judgment of the tongue and the identification of the tongue with tooth marks, because the fatness and thinness and the tongue with tooth marks need the lips as a reference to make the judgment easier. That is, we need to keep this relative relationship and discard more irrelevant background. Therefore, we expand the margin on the basis of the original bbox. Figure 5 This is the tongue image captured using a bbox with margin after segmentation. It can be seen that after the segmentation, the tongue occupies most of the image area, the background interference is greatly reduced, and the lip information and relative relationship are retained, which is conducive to improving the accuracy of subsequent classification and recognition; the single-side margin is generally set to 1 / 10 of the original bbox.
[0064] In addition to using the YOLO model, in other embodiments, models such as MaskRCNN and Unet may also be used.
[0065] Step S406, classifying the tongue images in each dimension.
[0066] A variety of CNN (convolutional neural network) classification models are used to classify the segmented tongue images in different dimensions, including tongue texture, tongue color, tongue shape, tongue coating, and tongue coating color.
[0067] First, the present invention uses the latest deep learning classification models, such as ResNet and YOLO cls for classification, to replace the traditional feature matching method. This replacement can significantly improve the accuracy in complex classification tasks. For example, current tongue diagnosis methods can rarely automatically identify the tongue spirit, because whether the tongue has a state cannot be described by a single feature, so it is difficult for traditional methods to accurately classify. The present invention uses a deep learning model. As long as the quality of the manually classified data set is guaranteed, the model can learn and acquire human experience, realize end-to-end learning, and even complex features can be accurately classified. Through this end-to-end method, the tongue spirit, tongue age, etc., which are difficult to automatically judge in existing tongue diagnosis, can be identified and classified.
[0068] Secondly, the present invention reclassifies the tongue images of traditional Chinese medicine tongue diagnosis to make it more suitable for computer logic. For example, for tongue shape, the traditional classification method includes fat tongue, thin tongue, old tongue, tender tongue, pricked tongue, tooth-marked tongue, and fissured tongue. This classification method has logical problems, and there may be overlaps between the classifications, which is not scientific enough. The present invention reclassifies the tongue shape into two dimensions: tongue shape-old and tender and tongue shape-fat and thin, and then divides it into old tongue and tender tongue in the old and tender dimension, and divides it into fat tongue and thin tongue in the fat and thin dimension. In this way, each dimension is single, atomic, and covers different dimensions. In addition, tooth-marked tongue, pricked tongue, and fissured tongue are used as target detection tasks to make judgments on their presence or absence. Similarly, in the classification of moss, the present invention re-divides more sub-dimensions, such as thin and thick, moist and dry, greasy and rotten, etc. By redefining the traditional Chinese medicine tongue diagnosis task, the present invention makes it more scientific, with independence and coverage in the classification dimension, and also makes it easier to produce annotated data sets, and the produced data sets are of higher quality.
[0069] The rearranged tongue classification is shown in the following table:
[0070]
[0071]
[0072] Step S408: local tongue image instance segmentation.
[0073] The YOLO instance segmentation model is used to detect and analyze local tongue images, including tooth-marked tongue, punctured tongue, and fissure tongue.
[0074] The traditional method for treating fissure tongue, tooth mark tongue and puncture tongue as tongue shape classification is to classify them, but this classification method will lead to classification confusion. In fact, these three types of tongue are actually whether the tongue has fissures, tooth marks and punctures. This application uses instance segmentation task area processing to detect whether there are these features and their specific areas. The advantage of this is that the classification of tongue shape is simpler, and at the same time, these three types of problems are more specifically located.
[0075] This application uses the trained and customized yolov8 model to perform instance segmentation on the fissure, puncture, and tooth mark areas of the tongue, and can accurately delineate the relevant areas, such as Figures 6 to 8 As shown, the crack area, puncture area, and tooth mark area are circled respectively;
[0076] These three tongue images are not necessarily present, and some people may not have any of these three tongue images; therefore, they should not be added as classification to other tongue shape dimensions; for example, other dimensions of tongue fatness, thinness, oldness, and tenderness are all independent tongue shape sub-dimensions, but these three tongue shapes are not. The present application embodiment will further detect the tongue based on the classification, so as to make the processing more accurate.
[0077] In addition, even if the traditional method can tell that it is a fissured tongue, there is no accurate area. The present application can accurately identify the specific area and the specific size of the area; therefore, the degree can be further quantified according to the number and size of the area to distinguish between mild and severe cases.
[0078] In summary, instance segmentation based on local tongue images is more effective in detecting fissure tongue, puncture tongue, and tooth-marked tongue than existing methods.
[0079] Step S410, assembling analysis prompts of different dimensions.
[0080] Integrate the classification results of each dimension and the analysis results of special tongue images to generate detailed analysis prompts. In the previous steps, the detection results of each dimension of tongue images were obtained. Next, these tongue images need to be further analyzed, that is, their meanings. Different dimensions represent different meanings, and different dimensions combined represent different meanings.
[0081] Existing solutions usually explain a single dimension, such as the meaning of a healthy tongue and the meaning of a dry tongue. However, it is usually difficult to maintain the meaning of a combination of multiple dimensions using rules, because there are too many possible combinations. For example, the classification provided in this application allows each dimension to combine 2,880 situations, which undoubtedly cannot be maintained using rules. Precisely because of this, existing solutions do not sort out so many dimensions and classifications, and try to ensure only the most basic classifications, and then easily organize a small number of rules.
[0082] In order to handle multiple dimensions and fine classifications, and then be able to interpret and analyze the meaning after tongue diagnosis, this application abandons the traditional rule-based method and instead uses a large language model to interpret, reason, and analyze. This application can not only explain the meaning of different tongue images, but also allow it to analyze more, such as constitution, symptoms, and corresponding conditioning plans.
[0083] For the use of the Large Language Model (LLM), the quality of the prompts is extremely important. The system prompts for the intelligent tongue diagnosis provided by this application are as follows: Fig. 9 As shown, it mainly includes 3 categories: 1) Constitution analysis based on tongue image: from tongue image to the analysis of 9 major constitutions, the user must be told what constitution he or she belongs to in the end; 2) Sign and symptom analysis based on tongue image: possible signs and symptoms are analyzed through tongue image, such as deducing internal heat based on red tongue; 3) Comprehensive analysis and conditioning plan suggestions: Comprehensive factors such as tongue spirit, tongue color, tongue shape, coating quality, and coating color are used to analyze and give a conditioning plan. The prompts of the large language model include system prompts and tongue image detection results. The system prompts are a standard prompt for the system. The specific system prompts are shown in the following table:
[0084]
[0085]
[0086] The quality of the prompt determines the quality of the large model output. As shown above, the system prompt of this application contains three parts: first, give the large model an identity, it is an old Chinese medicine doctor who is good at tongue diagnosis; second, tell it the specific analysis and reasoning ideas; finally, tell it the format to be output, such as json format, to facilitate subsequent processing; note that the prompt in the table above is only one possibility, as long as the general logic is the same, similar variants can achieve similar functions.
[0087] Step S412: The large model generates analysis results.
[0088] The integrated analysis prompts are input into a large model (such as GPT, Tongyi Qianwen, Llama3, etc.) for reasoning and analysis to generate comprehensive health status assessments and recommendations.
[0089] like Fig.10 As shown, the process of generating tongue image analysis by the large model of the present application specifically includes the following steps:
[0090] Step S4120, tongue image detection results.
[0091] Obtain tongue image detection results of various dimensions output by the aforementioned small models, such as classification models and instance segmentation models.
[0092] Step S4122, tongue image description reorganization.
[0093] Because the output of the previous small model is a result of a json structure, if it is directly stringified as the input of the large model, it is not friendly to the large model, because the naming of some fields in json may cause misunderstandings, and there are many irrelevant fields in the json body, resulting in too large input tokens, too high input costs, and insufficient focus of the large model, which is likely to be biased by irrelevant data. Therefore, it is necessary to reorganize the original tongue image recognition results. The reorganization method is to parse the json into a natural language, speak like a human, and feed it to the language large model. This can enhance the large model's understanding of the content, reduce the difficulty, improve the accuracy, and also reduce the number of tokens.
[0094] Step S4124, assembling prompts for each subtask.
[0095] As mentioned above, based on the tongue image detection results, the large language model should generate three subtasks: constitution analysis, symptom analysis, and comprehensive interpretation and suggestions. The system prompts mentioned above need to be assembled, and the natural language description reorganized in combination with the tongue image description in step S4126 should be used as user prompts and input into the large language model.
[0096] Step S4126, the large language model generates content simultaneously.
[0097] In order to speed up the process, an asynchronous concurrent method is used to call the large language model, so that subtasks can be sent to multiple large language models to generate different tongue image analyses.
[0098] Step S4128, data format verification and correction.
[0099] As mentioned above, in order to facilitate subsequent processing, the output of the large language model will be specified in the prompt as a fixed-format JSON body; however, the output of the large language model is not guaranteed to be output in the established format, so the output must be extracted, verified, and corrected. This application uses a large number of regular expressions for matching and verification.
[0100] Step S4129, finally, merge and output the tongue image analysis of each dimension, such as constitution analysis, symptom analysis, comprehensive interpretation and suggestions.
[0101] Step S414, result presentation.
[0102] The analysis results generated by the large model are displayed to the user, providing specific health status assessment and suggestions. The complete tongue analysis report provided by this application, for example Fig.11 As shown, including:
[0103] Health score. A scoring system based on the tongue test results; detailed later;
[0104] Constitution: Based on the comprehensive tongue image results, it is deduced that the user belongs to one of the nine constitutions, which include: peaceful constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-damp constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution;
[0105] Potential risks: the signs / symptoms mentioned above;
[0106] Tongue image analysis: identifies the specific tongue image on the tongue; provides schematic mode and real image mode; the schematic image is beautiful and not disgusting, making it easier for users to share tongue diagnosis reports and promote social dissemination of the program; the real image can see the details, and both modes are provided;
[0107] Tongue image explanation: Detailed explanation of each tongue image detected by the user, which is integrated with tongue image analysis;
[0108] Physical constitution analysis: inform the user in detail why he has that physical constitution;
[0109] Signs analysis: inform users in detail why they have certain signs / symptoms;
[0110] Comprehensive interpretation: includes detailed interpretation and suggestions for Chinese medicine conditioning plans.
[0111] Example 3
[0112] The present application also provides a method for calculating a health score based on tongue diagnosis, such as Fig.12 As shown, the method includes:
[0113] Step S1202: data collection and processing.
[0114] First, the tongue image data is collected and processed. The tongue diagnosis system is used to obtain various characteristics of the tongue, including color, shape, and coating.
[0115] Step S1204, classifying tongue image items.
[0116] Classify the tongue image data into different tongue image items. Assume that there are a total of A tongue image items detected, and each tongue image item has a corresponding deduction value Si and area ratio Pi. The specific tongue image items and their corresponding deduction values are as follows: the deduction value of a healthy tongue is 0; a dry tongue is -2; a pale tongue is -1; a pale red tongue is 0; a red tongue is -1; a crimson tongue is -1; a cyanotic tongue is -2; an old tongue is -2; a tender tongue is -1; a fat tongue is -1; a thin tongue is -1; a thin fur is -1; a thick fur is -2; a moist fur is -1; a dry fur is -2; a greasy fur is -1; a rotten fur is -2; a peeled fur is -2; a rooted fur is -1; a rootless fur is -2; a white fur is -1; a yellow fur is -2; a gray-black fur is -2.
[0117] Step S1206, area ratio calculation.
[0118] For tongue image items based on area ratio, such as fissure tongue, puncture tongue, tooth mark tongue, etc., determine its area ratio Pj. The rules for scoring the area ratio in stages are as follows: area ratio is 0: no points are deducted; area ratio <= 10%: 1 point is deducted; area ratio > 10%: 2 points are deducted.
[0119] Step S1208, calculating the health score.
[0120] The health score H of tongue diagnosis is calculated according to the following mathematical formula:
[0121]
[0122] Among them, Si is the deduction value of the i-th tongue image item, Pj is the area ratio of the j-th tongue image item based on area ratio, f(Pj) is the deduction function based on area ratio, and deduction is performed in different grades according to the area ratio Pj. n is the number of tongue image items with fixed deductions, and m is the number of tongue image items with deductions based on area ratio.
[0123] In another embodiment, the health score of tongue diagnosis can also be calculated according to the following formula:
[0124]
[0125]
[0126] Pj is the area ratio of the jth tongue image item based on area ratio, Wj is the severity weight of the jth tongue image item based on area ratio, Cj is the adjustment coefficient, and max(P) is the maximum value of all area ratios, which is used for normalization.
[0127] This application can more accurately reflect the impact of the tongue area ratio on the health score by introducing the adjustment coefficient Cj. The adjustment coefficient takes into account the area ratio and severity weight, making the scoring mechanism more accurate and detailed. In addition, the adjustment coefficient Cj introduces the severity weight Wj of the tongue image item, so that the impact of different tongue image items on the health score can be adjusted according to their severity. This can more accurately reflect the patient's health status. Finally, by using the maximum value of all area ratios for normalization, the adjustment coefficient Cj makes the deduction values of different tongue image items more comparable, avoiding the problem of inconsistent scoring due to different tongue area ratios.
[0128] Among them, the severity weight can be calculated based on the following formula:
[0129]
[0130] Among them, Rj is the health risk index of the jth tongue image item, Tj is the symptom duration of the jth tongue image item, Qj is the symptom frequency of the jth tongue image item, Sj is the symptom severity of the jth tongue image item, and Dj is a normalization constant used to ensure that the weight is within a reasonable range. α is a nonlinear adjustment coefficient used to adjust the nonlinear effect of the weight.
[0131] This application comprehensively considers the health risk index Rj, symptom duration Tj, symptom frequency Qj and symptom severity Sj, so that the scoring of each tongue image item is more comprehensive and scientific. Specifically, by introducing the health risk index Rj, the importance and risk of each tongue image item can be more accurately reflected, which ensures that the scoring does not only rely on a single factor, but comprehensively considers its medical and clinical importance. By introducing the symptom duration Tj and symptom frequency Qj, the scoring can dynamically reflect the changes in tongue image items. For example, tongue image items with long duration and high frequency will receive higher weights, thereby more accurately reflecting the patient's health status. In addition, the normalization constant Dj and the adjustment coefficient α provide flexibility, and the range of weights and nonlinear effects can be adjusted according to actual conditions.
[0132] The following is a specific calculation example. First, obtain the tongue image recognition results of different dimensions of the target object, and determine the multiple tongue image items detected based on the tongue image recognition results; assume that the user's fixed deduction tongue image items are dry tongue, thick fur, and rootless fur, and at the same time, assume that the user's tongue image items based on area ratio deduction data are as follows:
[0133]
[0134] Next, the deduction value of the tongue image items with fixed deduction points is calculated. In this embodiment, the tongue image items with fixed deduction points are dry tongue, thick fur, and fur without roots, each with a deduction value of 2 points, and the deduction value of the tongue image items with fixed deduction points is 6 points.
[0135] Calculate the deduction value of the tongue image item based on the area ratio deduction. Specifically, determine the severity weight Wj based on the health risk index, symptom duration, symptom frequency, symptom severity, and nonlinear adjustment coefficient; determine the adjustment coefficient based on the severity weight Wj, the area ratio of the tongue image item based on the area ratio, and the maximum value of all area ratios; determine the adjustment coefficient Cj based on the adjustment coefficient Cj and the grading function f(p j ) Determine the penalty value for tongue image items based on area ratio.
[0136] For example, to calculate the severity weight Wj:
[0137] Wj_crack tongue=(3*1.5*2*4 / 10)^1.2=4.87
[0138] W j_tooth mark tongue=(2.5*2*1.8*3.5 / 10)^1.2=4.36
[0139] Wj_Tongue pricked = (2*1*2.2*3 / 10)^1.2 = 1.54
[0140] 2) Calculate the adjustment coefficient Cj:
[0141] Cj_crack tongue=4.87*0.2 / 0.25=3.896
[0142] Cj_tooth mark tongue=4.36*0.15 / 0.25=2.616
[0143] Cj_Tongue pricked = 1.54*0.25 / 0.25 = 1.54
[0144] 3) Deduction value of tongue image items based on area ratio deduction
[0145] 3.896*2+2.616*2+1.54*2=16.104
[0146] Finally, the health score H is calculated based on the deduction values of the tongue image items with area ratio deductions and the deduction values of the tongue image items with fixed deductions.
[0147] H = 100 - (6 + 16.104) = 77.896
[0148] Example 4
[0149] The present application also provides a device for tongue analysis using an AI large model, such as Fig.13 As shown, it includes: a reorganization module 132, which is configured to obtain tongue image recognition results of different dimensions, and reorganize the tongue image recognition results of different dimensions to obtain corresponding natural language descriptions; a determination module 134, which is configured to determine the system prompts of the AI big model for different subtasks, wherein the different subtasks include a constitution analysis subtask, a symptom analysis subtask, and a comprehensive interpretation and suggestion subtask; an analysis module 136, which is configured to perform tongue image analysis using the AI big model in an asynchronous and concurrent manner based on the system prompts of each of the different subtasks and the corresponding natural language descriptions to obtain tongue image analysis results.
[0150] It should be noted that the device for tongue analysis using the AI big model provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device for tongue analysis using the AI big model provided in the above embodiment and the method embodiment for tongue analysis using the AI big model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0151] Example 5
[0152] Fig.14 FIG. 1 shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure. It should be noted that: Fig.14 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0153] like Fig.14 As shown, the electronic device includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0154] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.
[0155] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, various functions defined in the method and apparatus of the present application are executed. In some embodiments, the electronic device may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0156] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0157] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0158] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0159] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.
[0160] The computer-readable medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device can implement each step of the method embodiment.
[0161] As another aspect, the present application further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program executable by the at least one processor, and the computer program is used to enable the electronic device to perform the method of the embodiment of the present application when executed by the at least one processor.
[0162] As another aspect, the present application further provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.
[0163] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0164] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0165] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0166] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0167] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0169] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for tongue image analysis using AI big model, It is characterized in that include: Obtaining tongue image recognition results of different dimensions, and reorganizing the tongue image recognition results of different dimensions to obtain corresponding natural language descriptions; Determine the system prompts of the AI big model for different subtasks, wherein the different subtasks include constitution analysis subtask, symptom analysis subtask, and comprehensive interpretation and suggestion subtask; Based on the system prompts of each of the different subtasks and the corresponding natural language descriptions, the AI big model is used in an asynchronous and concurrent manner to perform tongue image analysis to obtain tongue image analysis results.
2. The method according to claim 1, It is characterized in that The tongue image recognition results of different dimensions are reorganized to obtain corresponding natural language descriptions, including: parsing the tongue image recognition results of different dimensions in the form of computer codes to obtain the natural language descriptions corresponding to the computer codes.
3. The method according to claim 1, It is characterized in that Determining system prompts for the AI big model for different subtasks includes: determining the system prompts for the different subtasks based on preset specifications, wherein the preset specifications include the identity of the analyst, the analyst's analysis and reasoning ideas, and the output format.
4. The method according to claim 1, It is characterized in that The AI big model is used in an asynchronous and concurrent manner to perform tongue analysis, including: Using the first AI big model in the AI big models to perform physical analysis on the physical analysis subtask, and generating an analysis result of the physical analysis dimension; Using the second AI big model in the AI big model to perform symptom analysis on the symptom analysis subtask, and generating analysis results of the symptom analysis dimension; Using the third AI big model in the AI big model to perform comprehensive interpretation and suggestion sub-analysis on the comprehensive interpretation and suggestion sub-task, and generate analysis results of the comprehensive dimension; The tongue image analysis result is generated based on the analysis result of the constitution analysis dimension, the analysis result of the symptom analysis dimension, and the analysis result of the comprehensive dimension.
5. The method according to claim 4, It is characterized in that In, including: Performing data format verification and correction on the analysis results of the constitution analysis dimension, the analysis results of the symptom analysis dimension, and the analysis results of the comprehensive dimension; The tongue image analysis result is obtained by merging the verified and corrected analysis results of the constitution analysis dimension, the symptom analysis dimension, and the comprehensive analysis result.
6. The method according to claim 1, It is characterized in that Obtain tongue image detection results in different dimensions, including: Acquire a tongue image of a target object, and perform segmentation processing on the tongue image using a segmentation model to separate the tongue region of the target object from the background region to obtain a tongue region image, wherein the tongue image includes the tongue region and the background region; Using a classification model to classify tongue images of different dimensions on the tongue region image to obtain tongue image classification results of different dimensions, and using an instance segmentation model to segment and analyze local tongue image instances on the tongue region image to obtain analysis results of local tongue images; The tongue image analysis result of the target object is generated based on the tongue image classification results of different dimensions and the analysis results of the local tongue image.
7. A device for tongue analysis using an AI big model, It is characterized in that include: a reorganization module, configured to obtain tongue image recognition results of different dimensions, and reorganize the tongue image recognition results of different dimensions to obtain corresponding natural language descriptions; A determination module is configured to determine system prompts of the AI big model for different subtasks, wherein the different subtasks include a constitution analysis subtask, a symptom analysis subtask, and a comprehensive interpretation and suggestion subtask; The analysis module is configured to perform tongue image analysis using the AI big model in an asynchronous and concurrent manner based on the system prompts of each of the different subtasks and the corresponding natural language descriptions to obtain tongue image analysis results.
8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.
9. A computer device, It is characterized in that include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor is enabled to execute the method according to any one of claims 1 to 6.
10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
