A tongue appearance health state analysis method, device, medium and electronic equipment

Abstract

The application relates to the technical field of tongue image analysis, and particularly provides a tongue image health state analysis method, device, medium and electronic equipment. The method can comprise the following steps: acquiring a cropped image of an original tongue image of a target object after cropping; performing image enhancement on the cropped image to obtain a plurality of to-be-detected images; performing tongue image classification on each to-be-detected image in the plurality of to-be-detected images to obtain a tongue image classification result of each to-be-detected image; the tongue image classification result comprises a plurality of regions in a plurality of dimensions on the tongue image; determining a tongue image health state based on a dimension weight distribution table and the tongue image classification result of each to-be-detected image; the tongue image health state is represented by a tongue image score and / or a tongue image health grade; the dimension weight distribution table comprises scores and weights of different regions under different dimensions of the tongue image. The embodiment of the application can realize comprehensive classification of the tongue image, and effectively improves the precision of tongue image health state evaluation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of tongue analysis, in particular to a tongue health state analysis method and device, a medium and an electronic device. BACKGROUND

[0002] Traditional tongue diagnosis is an important part of traditional Chinese medicine diagnosis. The color, shape, and coating of the tongue are observed to determine the health status of the human body.

[0003] Currently, traditional manual tongue diagnosis is based on the experience of the practitioner to give a diagnosis after observing the characteristics of the tongue. The accuracy of the diagnosis is difficult to guarantee. Alternatively, the tongue image of the target object is directly input into a pre-trained single deep learning model, and the deep learning model processes the tongue image and outputs the tongue classification result. However, the classification dimension of the tongue classification result is limited, and the specific situation of the tongue cannot be fully evaluated, and thus the accurate tongue health state analysis result cannot be obtained.

[0004] Therefore, how to provide a tongue health state analysis method with high accuracy becomes a technical problem to be solved. SUMMARY

[0005] Some embodiments of the present application aim to provide a tongue health state analysis method, device, medium, and electronic device. The technical scheme of the embodiments of the present application can improve the comprehensiveness of tongue classification, and thus obtain a tongue health state analysis result with high accuracy.

[0006] In a first aspect, some embodiments of the present application provide a tongue health state analysis method, comprising: obtaining a cropped image after the original tongue image of a target object is cropped; performing image enhancement on the cropped image to obtain a plurality of detection images; performing tongue classification on each detection image in the plurality of detection images to obtain a tongue classification result of each detection image; wherein the tongue classification result includes a plurality of regions in a plurality of dimensions on the tongue; determining a tongue health state based on a dimension weight distribution table and the tongue classification result of each detection image; wherein the tongue health state is represented by a tongue score and / or a tongue health level; and the dimension weight distribution table includes the scores and weights of different regions under different dimensions of the tongue.

[0007] Some embodiments of this application obtain multiple images to be detected by enhancing the cropped image; then, tongue image classification is performed on the multiple images to obtain tongue image classification results under multiple dimensions and multiple regions; finally, the tongue health status of the target object is determined by combining the dimension weight allocation table and the tongue image classification results. Embodiments of this application can achieve comprehensive and accurate tongue image classification, thereby obtaining highly accurate quantified tongue health status analysis results, providing doctors with reliable diagnostic basis.

[0008] In some embodiments, obtaining the cropped image after cropping the original tongue image of the target object includes: detecting the original tongue image to obtain a detection box; wherein the detection box includes a tongue region and a background region; expanding the detection box to obtain an expanded box; and using the expanded box to crop the original tongue image to obtain the cropped image.

[0009] Some embodiments of this application obtain detection boxes by detecting the original tongue image, and then expand the detection boxes to obtain expanded boxes to detect and crop the original tongue image to obtain a cropped image. This can obtain an image containing the tongue and background, thereby improving the robustness of color in subsequent tongue image classification.

[0010] In some embodiments, expanding the detection frame to obtain an expanded frame includes: expanding the detection frame according to a preset ratio to obtain the expanded frame; or, obtaining the current size of the detection frame; expanding it based on the expansion ratio corresponding to the current size to obtain the expanded frame; or, obtaining the confidence level of the detection frame; expanding it based on the expansion ratio corresponding to the confidence level to obtain the expanded frame.

[0011] Some embodiments of this application flexibly extend the detection box in a variety of different ways to support the automatic acquisition of standard cropped images.

[0012] In some embodiments, before performing image enhancement on the cropped image to obtain multiple images to be detected, the method further includes: inputting the cropped image into a pre-trained tongue coating classification model to obtain the probability value of the tongue image of the target object belonging to the tongue coating category.

[0013] Some embodiments of this application analyze cropped images using a tongue coating classification model to obtain probability values ​​for tongue coating categories, thereby achieving accurate identification of tongue coating, effectively distinguishing tongue coating from pathological tongue coating, improving diagnostic reliability, and avoiding misdiagnosis and overtreatment.

[0014] In some embodiments, the step of classifying the tongue image in each of the multiple images to be detected to obtain a tongue image classification result for each detected image includes: inputting each image to be detected into a tongue image classification model and outputting the tongue image classification result; wherein, the tongue image classification model is a pre-trained multi-task tongue image classification model, a multi-task network model, a cascaded network model, or a transformer network model; the multi-task network includes a tongue image feature classification layer, a dimensional classification layer, and a region classification layer.

[0015] Some embodiments of this application classify each image to be detected using a tongue image classification model, obtaining tongue image classification results in multiple dimensions and regions. The classification is relatively comprehensive, providing accurate support for subsequent tongue health status assessment.

[0016] In some embodiments, determining the tongue health status based on the dimensional weight allocation table and the tongue image classification results of each detected image includes: calculating the region probability of each of the multiple dimensions in the multiple regions based on the dimensional weight allocation table; weighting and summing the region probability of each dimension with the corresponding category score and normalizing the sum to obtain a normalized score for each dimension; multiplying the normalized score with the dimensional weight of each dimension to obtain a dimensional score for each dimension; calculating the dimensional score for each dimension to obtain the tongue image score; and determining the tongue health level corresponding to the tongue image score.

[0017] Some embodiments of this application assess the health status of the tongue image through a dimension weighting table and tongue image classification results, thereby achieving quantitative analysis of the tongue image, tracking the trend of changes in health status, and facilitating explanation and display to users.

[0018] In some embodiments, after determining the health status of the tongue, the method further includes: generating a tongue health report for the target object; wherein the tongue health report includes the probability value of the tongue coating category, the tongue classification result, or the tongue health status.

[0019] Some embodiments of this application can visually demonstrate the specific condition of the target object by generating a tongue health report.

[0020] Secondly, some embodiments of this application provide an apparatus for analyzing the health status of a tongue image, comprising: a cropping module for acquiring a cropped image of the original tongue image of a target object after cropping processing; an enhancement module for enhancing the cropped image to acquire multiple images to be detected; a classification module for classifying the tongue image of each of the multiple images to be detected to obtain a tongue image classification result for each detected image; wherein the tongue image classification result includes multiple regions in multiple dimensions of the tongue image; and a health assessment module for determining the health status of the tongue image based on a dimension weight allocation table and the tongue image classification result of each detected image; wherein the tongue image health status is represented by a tongue image score and / or a tongue image health level; the dimension weight allocation table includes the scores and weights of different regions of the tongue image in different dimensions.

[0021] Thirdly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0022] Fourthly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method as described in any embodiment of the first aspect.

[0023] Fifthly, some embodiments of this application provide a computer program product, the computer program product including a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of some embodiments of this application, the accompanying drawings used in some embodiments of this application will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A system diagram for tongue health status analysis provided for some embodiments of this application;

[0026] Figure 2 One of the flowcharts for tongue health status analysis provided in some embodiments of this application;

[0027] Figure 3 Flowchart 2 of the method for analyzing tongue health status provided for some embodiments of this application;

[0028] Figure 4 Block diagram of a device for analyzing tongue health status provided for some embodiments of this application;

[0029] Figure 5 A schematic diagram of an electronic device provided for some embodiments of this application. Detailed Implementation

[0030] The technical solutions of some embodiments of this application will now be described with reference to the accompanying drawings.

[0031] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] Among related technologies, one approach involves doctors visually observing the tongue and making a diagnostic conclusion based on their experience, judging characteristics such as tongue color, shape, coating texture, and coating color. This method heavily relies on the doctor's experience and subjective judgment. Another approach involves using simple image processing methods or a single deep learning model to analyze tongue images. This includes using traditional image processing techniques (such as color space conversion, threshold segmentation, edge detection, etc.) or neural networks (CNN, FNN, etc.) to analyze and classify tongue images (red tongue, fissured tongue, etc.).

[0033] However, image processing methods cannot accurately determine the tongue region; some image processing methods only segment the tongue and completely remove background information, which means completely ignoring the background information of the tongue. For example, the camera's automatic white balance and lighting environment can greatly affect color representation. A tongue photographed under yellow light and a tongue photographed under white light will have vastly different RGB values. If a portion of the background with known colors is retained (ideally including a standard color chart, or even the user's facial skin), the algorithm can use this to perform color correction and normalization on the entire image, thus obtaining a tongue color that is closer to reality and unaffected by ambient light. If the background is completely removed, for tasks centered on color recognition, blindly removing all background information is detrimental, and the final tongue image analysis results will be inaccurate. Furthermore, existing technologies have limited dimensions for classifying tongue images, and cannot comprehensively evaluate the specific situation of the tongue image; if a single inference method is used, the same image will show significant differences in performance under different lighting and angles, lacking robustness and resulting in low accuracy of tongue image analysis. Moreover, most existing technologies only output simple classification results and lack a quantitative evaluation system for tongue appearance, making it impossible to assess the health status of the tongue.

[0034] In view of this, some embodiments of this application provide a method for analyzing the health status of tongue images. This method involves image enhancement of the cropped image after processing the original tongue image to obtain multiple images to be detected; then, the multiple images to be detected are classified to obtain tongue image classification results under multiple dimensions and multiple regions; finally, the health status of the tongue image is determined through a dimension weight allocation table and the tongue image classification results. The embodiments of this application can achieve comprehensive classification of tongue images from multiple dimensions and multiple regions, and achieve accurate assessment of the health status of the tongue image. This method is highly reliable and intuitive, and is beneficial for health management and treatment efficacy evaluation.

[0035] The following is in conjunction with the appendix Figure 1 The overall structure of a system for analyzing tongue health status provided by some embodiments of this application is illustrated by way of example.

[0036] like Figure 1 As shown in the diagram, some embodiments of this application provide a system diagram for tongue health status analysis. This system may include a terminal 100 and a server 200. The terminal 100 can send the original tongue image of the target object to the server 200. The server 200 can crop and enhance the original tongue image to obtain multiple images to be detected; then, it performs tongue classification on the multiple images to obtain tongue classification results. Finally, it performs quantitative analysis on the tongue classification results according to a pre-set dimension weight allocation table to obtain the tongue health status.

[0037] In some embodiments of this application, the terminal 100 can be a mobile terminal or a non-portable computer terminal; no specific limitation is made here. Additionally, the server 200 is equipped with various pre-trained models related to tongue health status analysis to achieve accurate and comprehensive classification and health assessment of the tongue.

[0038] The following is in conjunction with the appendix Figure 2 The present application provides an exemplary embodiment of the implementation process of tongue health status analysis performed by server 200.

[0039] Please see the appendix Figure 2 , Figure 2 A flowchart of a method for analyzing the health status of a tongue image is provided for some embodiments of this application. The method for analyzing the health status of a tongue image may include:

[0040] S210, Obtain the cropped image after cropping the original tongue image of the target object.

[0041] For example, in a specific embodiment of this application, the original tongue image is cropped according to a set ratio to obtain a cropped image. Alternatively, a suitable cropping frame is defined for the original tongue image to crop it, resulting in a cropped image.

[0042] In some embodiments of this application, S210 may include: detecting the original tongue image to obtain a detection box; wherein the detection box includes a tongue region and a background region; expanding the detection box to obtain an expanded box; and using the expanded box to crop the original tongue image to obtain the cropped image.

[0043] For example, in a specific embodiment of this application, the YOLOv5 object detection network is used to automatically locate the tongue in the original tongue image and expand the background area to output a detection box. After the object detection network outputs the detection box, the size of the original detection box is expanded to ensure that the new detection box includes the entire tongue and a certain proportion of the background area, thus obtaining the expanded box; then the original tongue image is cropped according to the expanded box to obtain the cropped image.

[0044] Besides using the YOLOv5 object detection network for bounding box detection, other network models can be used, such as two-stage detection networks like Faster R-CNN and Mask R-CNN, or SSD (Single Shot MultiBoxDetector), RetinaNet, and DETR (Detection Transformer). Any network model capable of outputting bounding boxes and confidence scores for the tongue's location can replace the YOLOv5 object detection network. This application does not impose specific limitations on the embodiments described herein.

[0045] The detection box is expanded to obtain an expanded box, which can be achieved in the following three ways:

[0046] 1) Expand the detection frame according to a preset ratio to obtain the expanded frame.

[0047] For example, the width and height of the detection box are calculated, and the original detection box size is expanded by a certain expansion ratio (as a specific example of a preset ratio, for example, 25%) to obtain an expanded box. The expansion ratio can be any value; it can be set according to the actual application scenario, and the embodiments of this application are not limited to this.

[0048] 2) Obtain the current size of the detection frame; expand it based on the expansion ratio corresponding to the current size to obtain the expanded frame.

[0049] For example, the size of the detection frame is adaptively and dynamically adjusted based on its size to obtain an expanded frame. For instance, for small detection frames smaller than 200×200, the expansion ratio can be 30%; for medium detection frames between 200×200 and 400×400, the expansion ratio can be 25%; and for large detection frames larger than 400×400, the expansion ratio can be 20%. It is understood that the correspondence between the size of the detection frame and the expansion ratio can be flexibly adjusted, and the embodiments of this application are not limited to this.

[0050] 3) Obtain the confidence level of the detection box; expand the detection box based on the expansion ratio corresponding to the confidence level to obtain the expanded box.

[0051] For example, the detection box can be dynamically adjusted adaptively based on the confidence level. For instance, when the confidence level of the detection box is greater than 0.9, the expansion ratio can be 20%; when the confidence level is between 0.5 and 0.9, the expansion ratio can be 25%; and when the confidence level is less than 0.5, the expansion ratio can be 30%. It is understood that the correspondence between the confidence level of the detection box and the expansion ratio can be flexibly adjusted, and the embodiments of this application are not limited thereto.

[0052] In addition to the methods mentioned above, other methods can be used to determine the detection frame, and the embodiments of this application do not specifically limit them here.

[0053] Since existing tongue diagnosis systems lack the function of judging tongue coating, in some embodiments of this application, before executing S220, the method for analyzing the health status of the tongue image may further include: inputting the cropped image into a pre-trained tongue coating classification model to obtain the probability value of the tongue image of the target object belonging to the tongue coating category.

[0054] For example, in a specific embodiment of this application, a dedicated moss classification module is introduced. The moss classification module uses the ResNet18 network structure (as a specific example of a moss classification model) to extract features and classify the cropped image, and outputs a moss category probability value.

[0055] If the probability value of the tongue coating category is greater than a set threshold (e.g., 0.86), it is considered a stained tongue coating; otherwise, it is considered a non-stained tongue coating. The set threshold can be any value between 0.1 and 1, and can be adaptively adjusted according to actual conditions. Alternatively, the optimal preset threshold can be determined statistically based on a dataset of tongue coating related to tongue appearance. Or, the optimal preset threshold can be determined using an ROC curve.

[0056] Alternatively, a multi-threshold strategy can be used to determine whether a tongue coating is stained. For example, if the probability value of the stained tongue coating category is greater than 0.9, it is determined to be stained tongue coating; if the probability value of the stained tongue coating category is within (0.7, 0.9], it may be stained tongue coating, and a follow-up examination will be suggested in the subsequent tongue health report; if the probability value of the stained tongue coating category is less than or equal to 0.7, it is determined to be non-stained tongue coating.

[0057] It is understandable that the thresholds set above can be adaptively adjusted according to the sensitivity and specificity requirements of the application scenario.

[0058] In addition to using the ResNet18 network structure, you can also use deeper ResNet series such as ResNet50 and ResNet101; EfficientNet series (B0-B7); MobileNet series (lightweight solution); VGGNet, DenseNet; Vision Transformer (ViT); Swin Transformer, etc. Any image classification network that can achieve binary classification can be used.

[0059] It should be noted that the classification of tongue coating and subsequent tongue appearance feature classification operate independently. Failure in the tongue coating classification does not affect the tongue appearance classification, and the two results can be integrated into the final report. This module can effectively distinguish between tongue coating and pathological tongue coating, improve diagnostic reliability, and avoid misdiagnosis and overtreatment.

[0060] S220, perform image enhancement on the cropped image to obtain multiple images to be detected.

[0061] For example, in a specific embodiment of this application, multiple images to be detected are obtained through image enhancement for subsequent batch inference, thereby improving the robustness of classification.

[0062] Image enhancement can be achieved in several ways, including the following:

[0063] 1) Rotate the cropped image at multiple angles and adjust its brightness to generate a total of 4 images (as a specific example of multiple images to be detected); these 4 images may include the cropped image and the three images obtained by rotating it by 90°, 180° and 270°.

[0064] 2) The four images obtained by flipping the cropped image from multiple angles are the cropped image and three images obtained by horizontal flipping, vertical flipping, and horizontal and vertical flipping.

[0065] 3) Only use the cropped image and the two images rotated 180°; as well as the images corresponding to the single-image mode and quick mode of the cropped image.

[0066] 4) Perform a quality assessment on the cropped image to determine whether enhancement is necessary. If it is a high-quality image (quality score greater than the quality threshold), no enhancement is required; otherwise, image enhancement is required.

[0067] It is understood that the method and amount of image enhancement can be adjusted according to the trade-off between accuracy and speed, and the embodiments of this application do not impose specific limitations here.

[0068] Image augmentation can simulate different shooting angles and lighting conditions, thereby improving the model's robustness to changes in angle and lighting. Subsequent multi-image inference can also improve classification accuracy, reduce the risk of misjudgment, and make the results more stable and reliable.

[0069] S230, perform tongue image classification on each of the multiple images to be detected to obtain the tongue image classification result for each detected image; wherein, the tongue image classification result includes multiple regions in multiple dimensions of the tongue image.

[0070] For example, in a specific embodiment of this application, batch reasoning is performed on the four images obtained above to obtain the tongue image classification result for each image. Compared with the prior art, which can only analyze a few dimensions and does not distinguish regions, this application can achieve a tongue image classification result of 65 categories with 13 dimensions and 5 regions (as a specific example of multiple dimensions and multiple regions). The specific contents of these 65 categories are shown in Table 1. The five regions in the following categories conform to traditional Chinese medicine theory, such as the tip of the tongue reflecting the heart and lungs, and the root of the tongue reflecting the kidneys.

[0071] Table 1

[0072]

[0073] The aforementioned 65 categories can be adjusted as needed (e.g., adding or deleting related regions, dimensions, and categories). As long as the task design of regional, multi-dimensional, and multi-category classification is met, the specific regional, dimension, and category design can be adjusted according to the professional content of traditional Chinese medicine. This application's embodiments are not limited to this.

[0074] In some embodiments of this application, S230 may include: inputting each image to be detected into a tongue image classification model and outputting the tongue image classification result; wherein, the tongue image classification model is a pre-trained multi-task tongue image classification model, a multi-task network model, a cascaded network model, or a transformer network model; the multi-task network includes a tongue image feature classification layer, a dimensional classification layer, and a region classification layer.

[0075] For example, in a specific embodiment of this application, ResNet is used as the backbone network (as a specific example of a multi-task tongue image classification model) to process the multiple images to be detected obtained above, extract image features, and output 65 classification task results (as a specific example of tongue image classification results). The analysis dimensions are comprehensive, covering all aspects of TCM tongue diagnosis. This multi-task tongue image classification model is set with a shared feature extraction layer (i.e., a shared layer) and 65 independent classification heads (i.e., task heads); in addition, an attention module can be added between the shared layer and the task heads to automatically learn the feature weights of different tasks.

[0076] In addition to using a multi-task tongue image classification model, a multi-task network model can also be used. This multi-task network model consists of three layers. The first layer is a two-branch classification of tongue body features and tongue coating features. The second layer further divides each of these two branches into 13 dimensions. The third layer further divides each dimension into 5 regions, thereby outputting the tongue image classification results.

[0077] If a cascaded network model is used, it first analyzes the overall features of the image, then analyzes the regional features based on the overall features, and outputs the tongue image classification result. Alternatively, a Transformer structure (i.e., a transformer network model) can be used, specifically using a Transformer encoder to replace the CNN feature extraction layer and employing a multi-head attention mechanism to handle multiple tasks.

[0078] It is understood that tongue image classification models can be adaptively adjusted according to specific application scenarios, and the embodiments of this application are not specifically limited here.

[0079] S240, Based on the dimensional weight allocation table and the tongue image classification results of each detected image, determine the tongue image health status; wherein, the tongue image health status is represented by tongue image score and / or tongue image health level; the dimensional weight allocation table includes the score and weight of different regions of the tongue image under different dimensions.

[0080] For example, in a specific embodiment of this application, a dimension weight allocation table designed by a professional clinical TCM doctor is used. Combined with the tongue appearance classification results and the probability distribution of the 13 dimensions within the table, the scores of each dimension are accumulated to obtain a comprehensive percentage-based health score, and the level is confirmed, thereby quantitatively assessing the health status of the tongue appearance. The dimension weight allocation table includes a 13-dimensional weight allocation table and a linear score mapping algorithm table for each dimension. The 13-dimensional weight allocation table is shown in Table 2.

[0081] Table 2

[0082]

[0083] Taking tongue color as an example, the linear score mapping algorithm table is shown in Table 3 (i.e., an example of the mapping between 10 tongue color categories and the original score):

[0084] Table 3

[0085]

[0086] In addition to using a linear score mapping table as described above, nonlinear score mapping rules or multi-dimensional special rules can also be used. For example, if teeth marks are combined with tongue swelling / full mouth, a score of -0.2 can be assigned. The specific choice can be made according to the actual situation, and this application does not impose specific limitations on the embodiments herein.

[0087] In some embodiments of this application, S240 may include: calculating the regional probability of each of the plurality of dimensions under the plurality of regions based on the dimensional weight allocation table; weighting and summing the regional probability under each dimension with the corresponding category score and normalizing the sum to obtain a normalized score for each dimension; multiplying the normalized score with the dimensional weight of each dimension to obtain a dimensional score for each dimension; calculating the dimensional score for each dimension to obtain the tongue image score; and determining the tongue image health level corresponding to the tongue image score.

[0088] For example, in a specific embodiment of this application, the regional probabilities of five regions under each of the 13 dimensions in the tongue image classification result are fused. This fusion can be achieved by averaging the probability values ​​of the five regions, or by using a weighted average (e.g., a surface weight of 0.4 and other regions weights of 0.15). Alternatively, a voting mechanism can be used, where each of the five regions predicts its own category, and the probability corresponding to the category with the most occurrences is selected as the regional probability. Alternatively, the prediction result with the highest probability among the five regions can be used as the regional probability. Alternatively, the probability can be obtained by weighting the five regions according to their confidence levels, where regions with high confidence levels have larger weights. Specifically, the fusion method for the five regions can be selected based on the characteristics of different dimensions; this embodiment of the application does not impose specific limitations on this.

[0089] For example, taking tongue color as an example, after fusion, the resulting regional probability distribution is: {'pale red tongue': 0.6, 'pale white tongue': 0.3, 'red tongue': 0.1}.

[0090] After the above fusion is completed, for each dimension, the weighted sum is performed using the fused probability distribution: weighted_score = Σ(original score[category] × region probability[category]).

[0091] For example, as shown in Table 3, the original score is: weighted_score = 1.0×0.6 + 0.4×0.3 + 0.4×0.1 = 0.6 + 0.12 + 0.04 = 0.76.

[0092] Normalize the weighted_score using the following formula: normalized_score = (weighted_score + 0.3) / 1.3, which gives normalized_score = (0.76 + 0.3) / 1.3 = 0.815 (as a specific example of a normalized score).

[0093] Finally, the score for the tongue color dimension is obtained: dimension_score = 0.815×15 = 12.2 points (as a specific example of dimension score).

[0094] By following the above method, scores can be obtained for different dimensions. Finally, these scores are added together to obtain the scores for the tongue body and tongue coating respectively. For example, the tongue body scores 45 points and the tongue coating scores 55 points. The sum of the two scores gives the tongue appearance score.

[0095] By using a pre-constructed mapping table between tongue image scores and tongue image health levels, the corresponding tongue image health level can be obtained. For example, tongue image health levels can be divided into qualitative health levels such as healthy, sub-healthy, and mildly abnormal.

[0096] In some embodiments of this application, after S240 is executed, the method for analyzing the health status of the tongue image may further include: generating a tongue health report for the target object; wherein the tongue health report includes the probability value of the coating category, the tongue image classification result, or the tongue health status.

[0097] For example, in a specific embodiment of this application, a corresponding tongue health report can be generated based on one or more of the above-mentioned probability values ​​of tongue coating categories, tongue image classification results, and tongue health status. This tongue health report can group all categories according to tongue body characteristics and tongue coating characteristics, separate normal and abnormal dimensions, simplify the display of normal dimensions, provide detailed information on abnormal dimensions, and finally output in a structured JSON format. Example of output results:

[0098] "Tongue characteristics": {

[0099] "Normal dimensions": "The tongue is generally light red, of moderate thickness and size, without cracks or teeth marks";

[0100] "Abnormal Dimension": "Petechiae and Ecchymoses: Pinillitis on the tongue (Confidence: 0.75)"

[0101] },

[0102] "Characteristics of tongue coating": {

[0103] "Normal dimensions": "The tongue coating is thin, white, moist, evenly distributed, and has normal granules";

[0104] "Abnormal Dimension": ""}.

[0105] Besides JSON format, output can also be in XML format, Protobuf format, and more. It can also generate natural language reports, i.e., complete textual descriptions using natural language processing technology. Alternatively, it can generate visual reports, such as charts, heatmaps, or feature annotations on tongue images; or it can use a tiered output approach, for example, a simplified version outputting only the total score and health level; a standard version including the main dimensions; and a detailed version including all 65 dimensions. Understandably, the output format can be customized according to different application scenarios and user needs.

[0106] The following is in conjunction with the appendix Figure 3 The present application provides an exemplary description of the specific process of tongue health status analysis provided by some embodiments.

[0107] Please see the appendix Figure 3 , Figure 3 A flowchart illustrating a method for analyzing the health status of a tongue image, provided for some embodiments of this application.

[0108] The above process is illustrated below by example.

[0109] S310: Detect the original tongue image and obtain the detection box.

[0110] S320, expand the detection box to obtain an expanded box.

[0111] S330: Use an expanded frame to crop the original tongue image to obtain the cropped image.

[0112] S340, input the cropped image into the pre-trained tongue coating classification model to obtain the probability value of the tongue image of the target object belonging to the tongue coating category.

[0113] S350 performs image enhancement on the cropped image to obtain multiple images to be detected.

[0114] S360 inputs each image to be detected into the tongue image classification model and outputs the tongue image classification result.

[0115] S370 determines the health status of the tongue image based on the dimension weight allocation table and the tongue image classification results of each detected image.

[0116] S380, Generate a tongue health report for the target object.

[0117] The tongue health report includes the probability value of tongue coating category, tongue classification results, and tongue health status.

[0118] It should be noted that the specific implementation process of S310 to S380 can be referred to the method embodiment provided above. To avoid repetition, detailed descriptions are appropriately omitted here. Among them, the steps after S340 and S350 can be executed in parallel or in a certain order. This application embodiment does not make specific limitations here.

[0119] As can be seen from the above embodiments of this application, this application uses a YOLO object detection network + adaptive expansion algorithm for target determination. The YOLOv5 network, trained on a large amount of tongue image data, can accurately identify the tongue position with a detection accuracy >95%. The adaptive expansion algorithm expands the detection box according to a preset ratio, preserving complete tongue information while avoiding excessive background interference. Compared with existing background segmentation methods, this solution is more adaptable to different backgrounds, lighting, angles, and bright color paths in photography, and is fully automated, requiring no manual annotation, thus significantly improving efficiency.

[0120] Furthermore, this application is the first to achieve intelligent judgment of tongue coating staining, improving diagnostic reliability. It introduces a ResNet18 tongue coating staining binary classification model combined with adaptive threshold judgment. Existing tongue diagnosis systems generally lack tongue coating staining judgment, easily misclassifying stained tongue coating as pathological tongue coating. This application, through a specialized binary classification model, accurately identifies stained tongue appearance with an accuracy rate >92%. The adaptive threshold has been verified through extensive experiments, effectively balancing sensitivity and specificity, avoiding misdiagnosis caused by tongue coating staining, and improving diagnostic reliability and clinical application value.

[0121] This application utilizes a multi-task learning network and a five-region joint analysis design, employing 65 classification tasks to cover 13 core dimensions of TCM tongue diagnosis, making it more comprehensive than existing systems. The five-region analysis (tip, root, sides, center, and overall) aligns with TCM organ location theory (the tip reflects the heart and lungs: a red tip is common when heart fire is excessive; the root reflects the kidneys: a thick, greasy coating is common when kidney deficiency is present; the sides reflect the liver and gallbladder: a red side is common when liver stagnation transforms into fire; the center reflects the spleen and stomach: a thick coating is common when spleen deficiency is accompanied by dampness). Through a probabilistic fusion algorithm, information from the five regions is integrated to arrive at an overall assessment result, achieving refined and personalized health evaluation.

[0122] Furthermore, in the multi-angle rotation + batch inference + probability averaging algorithm, multi-angle rotation simulates different shooting angles, improving the model's robustness; inference is performed on four enhanced images separately, obtaining four sets of prediction results; through the probability averaging algorithm, the final probability of each category is the average of the probabilities from the four inferences. Averaging multiple measurements reduces random errors. Compared to single inference, the accuracy is improved by 5-10%, and the results are more stable and reliable. The improvement is particularly significant for tongue images with unclear boundaries and indistinct features. In short, the image enhancement + probability averaging method of this application exhibits high accuracy and stability.

[0123] Finally, this application improves the quantitative assessment system to facilitate health management. Specifically, the scoring algorithm incorporates expert knowledge for tiered assessment. Different dimensions are assigned different weights based on their importance in Traditional Chinese Medicine (TCM) theory (e.g., tongue color 15 points, coating thickness 12 points, etc.); non-linear scoring mapping reflects the severity of different categories (e.g., "pale red tongue" receives full marks, "purple spots" receive negative marks); special association rules (e.g., combined assessment of teeth marks and tongue swelling) incorporate TCM diagnostic thinking; the 100-point system is intuitive and easy to understand, and multiple health levels facilitate tiered management; detailed dimensional scoring supports targeted health interventions and efficacy tracking. Compared to existing systems that only output classification results, this solution provides users with more valuable information.

[0124] Please refer to Figure 4 , Figure 4 The diagram shows a block diagram of a tongue health status analysis apparatus provided in some embodiments of this application. It should be understood that this tongue health status analysis apparatus corresponds to the method embodiments described above and is capable of performing the various steps involved in the method embodiments. The specific functions of this tongue health status analysis apparatus can be found in the description above; detailed descriptions are omitted here to avoid repetition.

[0125] Figure 4The device for analyzing the health status of tongue images includes at least one software functional module that can be stored in a memory or embedded in the device in the form of software or firmware. The device includes: a cropping module 410 for acquiring a cropped image of the original tongue image of the target object; an enhancement module 420 for enhancing the cropped image to acquire multiple images to be detected; a classification module 430 for classifying the tongue image of each of the multiple images to be detected, obtaining a tongue image classification result for each detected image; wherein the tongue image classification result includes multiple regions in multiple dimensions of the tongue image; and a health assessment module 440 for determining the health status of the tongue image based on a dimension weight allocation table and the tongue image classification result of each detected image; wherein the tongue image health status is represented by a tongue image score and / or a tongue image health level; the dimension weight allocation table includes the scores and weights of different regions of the tongue image in different dimensions.

[0126] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0127] Some embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can perform the operation of any of the methods corresponding to the methods provided in the above embodiments.

[0128] Some embodiments of this application also provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operation of any of the methods corresponding to the above embodiments provided in the above embodiments.

[0129] like Figure 5 As shown, some embodiments of this application provide an electronic device 500, which includes a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520. When the processor 520 reads the program from the memory 510 via a bus 530 and executes the program, it can implement the methods of any of the above embodiments.

[0130] Processor 520 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 520 can be a microprocessor.

[0131] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 520 of this disclosure embodiment can be used to execute the instructions in the memory 510 to implement the methods shown above. The memory 510 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0132] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0133] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0134] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for analyzing the health status of the tongue, characterized in that, include: Obtain the cropped image after cropping the original tongue image of the target object; Image enhancement is performed on the cropped image to obtain multiple images to be detected; Tongue image classification is performed on each of the multiple images to be detected to obtain the tongue image classification result for each detected image; wherein, the tongue image classification result includes multiple regions in multiple dimensions of the tongue image; the division of the regions conforms to traditional Chinese medicine theory; Based on the dimensional weight allocation table and the tongue image classification results of each detected image, the health status of the tongue image is determined; wherein, the health status of the tongue image is represented by the tongue image score and / or tongue image health level; the dimensional weight allocation table includes the score and weight of different regions of the tongue image under different dimensions; The determination of tongue health status based on the dimensional weight allocation table and the tongue image classification results for each detected image includes: Based on the dimension weight allocation table, calculate the regional probability of each of the multiple dimensions in the multiple regions; The region probability under each dimension is weighted and summed with the corresponding category score, and then normalized to obtain the normalized score for each dimension. The normalized score is multiplied by the dimensional weight of each dimension to obtain the dimensional score of each dimension. The score for each dimension is calculated to obtain the tongue image score; and the tongue image health level corresponding to the tongue image score is determined. Each dimension corresponds to a linear score mapping algorithm table; the linear score mapping algorithm table includes class name and score; the weighted summation and normalization of the region probability and corresponding category score under each dimension to obtain the normalized score of each dimension includes: The category score corresponding to the region probability is determined based on the linear score mapping algorithm table; The original score is obtained by multiplying the probability of each region by its corresponding category score and then summing the results. The original scores are normalized to obtain the normalized scores for each dimension; The step of classifying the tongue image in each of the multiple images to be detected, and obtaining the tongue image classification result for each detected image, includes: Each image to be detected is input into a tongue image classification model, which outputs the tongue image classification result. The tongue image classification model is a pre-trained multi-task network model. The multi-task network includes a tongue image feature classification layer, a dimensional classification layer, and a region classification layer. The tongue image feature classification layer is a two-branch classification of tongue body features and tongue coating features. The dimensional classification layer further divides each branch of the two-branch classification into 13 dimensions. The region classification layer further divides each dimension into 5 regions. The 5 regions include the surface of the tongue, the tip of the tongue, the root of the tongue, the sides of the tongue, and the middle of the tongue. The calculation of the dimension score for each dimension to obtain the tongue image score includes: Based on the dimensional score of each dimension, a non-linear scoring mapping rule is used to calculate the dimensional scores of multiple dimensions to obtain the tongue image score.

2. The method according to claim 1, characterized in that, The cropped image obtained after cropping the original tongue image of the target object includes: The original tongue image is detected to obtain a detection box; wherein the detection box includes a tongue region and a background region; The detection box is expanded to obtain an expanded box; The original tongue image is cropped using the expanded frame to obtain the cropped image; The cropped image is color-corrected and normalized based on the color of the background region.

3. The method as described in claim 2, characterized in that, The step of expanding the detection box to obtain an expanded box includes: The detection frame is expanded according to a preset ratio to obtain the expanded frame; or... Obtain the current size of the detection frame; expand it based on the expansion ratio corresponding to the current size to obtain the expanded frame; or, Obtain the confidence level of the detection box; expand the detection box based on the expansion ratio corresponding to the confidence level to obtain the expanded box.

4. The method according to any one of claims 1-3, characterized in that, Before performing image enhancement on the cropped image to obtain multiple images to be detected, the method further includes: The cropped image is input into a pre-trained tongue coating classification model to obtain the probability value of the tongue image of the target object belonging to the tongue coating category.

5. The method as described in claim 4, characterized in that, After determining the health status of the tongue image, the method further includes: Generate a tongue health report for the target object; wherein the tongue health report includes the probability value of the tongue coating category, the tongue classification result, or the tongue health status.

6. A device for analyzing the health status of the tongue, characterized in that, include: The cropping module is used to obtain the cropped image after cropping the original tongue image of the target object; The enhancement module is used to enhance the cropped image and obtain multiple images to be detected; The classification module is used to classify the tongue image in each of the multiple images to be detected, and obtain the tongue image classification result for each detected image; wherein, the tongue image classification result includes multiple regions in multiple dimensions of the tongue image; The health assessment module is used to determine the health status of the tongue image based on the dimensional weight allocation table and the tongue image classification results of each detected image; wherein, the tongue image health status is represented by tongue image score and / or tongue image health level; the dimensional weight allocation table includes the score and weight of different regions of the tongue image under different dimensions. The health assessment module is specifically used for: Based on the dimension weight allocation table, calculate the regional probability of each of the multiple dimensions in the multiple regions; The region probability under each dimension is weighted and summed with the corresponding category score, and then normalized to obtain the normalized score for each dimension. The normalized score is multiplied by the dimensional weight of each dimension to obtain the dimensional score of each dimension. The score for each dimension is calculated to obtain the tongue image score; and the tongue image health level corresponding to the tongue image score is determined. Each dimension corresponds to a linear score mapping algorithm table; the linear score mapping algorithm table includes class name and score; the weighted summation and normalization of the region probability and corresponding category score under each dimension to obtain the normalized score of each dimension includes: The category score corresponding to the region probability is determined based on the linear score mapping algorithm table; The original score is obtained by multiplying the probability of each region by its corresponding category score and then summing the results. The original scores are normalized to obtain the normalized scores for each dimension; The step of classifying the tongue image in each of the multiple images to be detected, and obtaining the tongue image classification result for each detected image, includes: Each image to be detected is input into a tongue image classification model, which outputs the tongue image classification result. The tongue image classification model is a pre-trained multi-task network model. The multi-task network includes a tongue image feature classification layer, a dimensional classification layer, and a region classification layer. The tongue image feature classification layer is a two-branch classification of tongue body features and tongue coating features. The dimensional classification layer further divides each branch of the two-branch classification into 13 dimensions. The region classification layer further divides each dimension into 5 regions. The 5 regions include the surface of the tongue, the tip of the tongue, the root of the tongue, the sides of the tongue, and the middle of the tongue. The calculation of the dimension score for each dimension to obtain the tongue image score includes: Based on the dimensional score of each dimension, a non-linear scoring mapping rule is used to calculate the dimensional scores of multiple dimensions to obtain the tongue image score.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to perform the method as described in any one of claims 1-5.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the computer program is executed by the processor to perform the method as described in any one of claims 1-5.

Images