Tongue picture health state analysis method and device, medium and electronic equipment

Abstract

The invention relates to the technical field of tongue picture analysis, and particularly provides a tongue picture health state analysis method and device, a medium and electronic equipment, and the method can comprise the steps: obtaining a cut image obtained by cutting an original tongue picture image of a target object; performing image enhancement on the cut image to obtain a plurality of to-be-detected images; performing tongue picture classification on each to-be-detected image in the plurality of to-be-detected images to obtain a tongue picture classification result of each to-be-detected image; the tongue picture classification result comprises a plurality of areas in a plurality of dimensions on the tongue picture; determining a tongue picture health state based on the dimension weight distribution table and the tongue picture classification result of each detection image; the tongue picture health state is represented by a tongue picture score and / or a tongue picture health level; the dimension weight distribution table comprises scores and weights of the tongue picture in different areas under different dimensions. According to the embodiment of the invention, comprehensive classification of the tongue pictures can be realized, and the accuracy of tongue picture health state evaluation is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of tongue analysis, in particular to a tongue health state analysis method and device, a medium and an electronic device. BACKGROUND

[0002] Traditional tongue diagnosis is an important part of traditional Chinese medicine diagnosis. The color, shape, and coating of the tongue are observed to determine the health status of the human body.

[0003] Currently, traditional manual tongue diagnosis is based on the experience of the practitioner to give a diagnosis after observing the characteristics of the tongue. The accuracy of the diagnosis is difficult to guarantee. Alternatively, the tongue image of the target object is directly input into a pre-trained single deep learning model, and the deep learning model processes the tongue image and outputs the tongue classification result. However, the classification dimension of the tongue classification result is limited, and the specific situation of the tongue cannot be fully evaluated, and thus an accurate tongue health state analysis result cannot be obtained.

[0004] Therefore, how to provide a tongue health state analysis method with high accuracy becomes a technical problem to be solved. SUMMARY

[0005] Some embodiments of the present application aim to provide a tongue health state analysis method, device, medium, and electronic device. The technical solutions of the embodiments of the present application can improve the comprehensiveness of tongue classification, and thus obtain a tongue health state analysis result with high accuracy.

[0006] In a first aspect, some embodiments of the present application provide a tongue health state analysis method, comprising: obtaining a cropped image after the original tongue image of a target object is cropped; performing image enhancement on the cropped image to obtain a plurality of detection images; performing tongue classification on each detection image in the plurality of detection images to obtain a tongue classification result of each detection image; wherein the tongue classification result includes a plurality of regions in a plurality of dimensions on the tongue; determining a tongue health state based on a dimension weight distribution table and the tongue classification result of each detection image; wherein the tongue health state is represented by a tongue score and / or a tongue health level; and the dimension weight distribution table includes the scores and weights of different regions under different dimensions of the tongue.

[0007] Some embodiments of the present application obtain multiple to-be-detected images by enhancing the cropped image; then perform tongue image classification on the multiple to-be-detected images to obtain tongue image classification results in multiple dimensions and multiple regions; and finally determine the tongue health status of the target object in combination with the dimension weight distribution table and the tongue image classification results. The embodiments of the present application can realize comprehensive and accurate classification of tongue images, and thus obtain a quantitative tongue health status analysis result with high accuracy, thereby providing a reliable diagnosis basis for doctors.

[0008] In some embodiments, the cropped image obtained by cropping the original tongue image of the target object comprises: detecting the original tongue image to obtain a detection frame; wherein the detection frame comprises a tongue body region and a background region; expanding the detection frame to obtain an expansion frame; and cropping the original tongue image using the expansion frame to obtain the cropped image.

[0009] Some embodiments of the present application obtain a detection frame by detecting an original tongue image, and then obtain a cropped image by cropping the original tongue image using an expansion frame obtained by expanding the detection frame. In this way, an image containing a tongue body and a background can be obtained, thereby improving the robustness of color in subsequent tongue image classification.

[0010] In some embodiments, the expansion of the detection frame to obtain an expansion frame comprises: expanding the detection frame according to a preset expansion ratio to obtain the expansion frame; or obtaining a current size of the detection frame; expanding based on an expansion ratio corresponding to the current size to obtain the expansion frame; or obtaining a confidence of the detection frame; expanding based on an expansion ratio corresponding to the confidence to obtain the expansion frame.

[0011] Some embodiments of the present application flexibly expand the detection frame in multiple different ways to provide support for automatically obtaining a standard cropped image.

[0012] In some embodiments, before the image enhancement of the cropped image to obtain multiple to-be-detected images, the method further comprises: inputting the cropped image into a pre-trained dyeing classification model to obtain a dyeing category probability value of the tongue image of the target object belonging to dyeing.

[0013] Some embodiments of the present application obtain a dyeing category probability value by analyzing the cropped image using a dyeing classification model, thereby realizing accurate identification of dyeing, effectively distinguishing between dyeing and pathological tongue fur, improving diagnosis reliability, and avoiding misdiagnosis and over-treatment.

[0014] In some embodiments, the tongue image classification of each of the plurality of to-be-detected images to obtain a tongue image classification result of each of the plurality of to-be-detected images comprises: inputting each of the plurality of to-be-detected images into a tongue image classification model to output the tongue image classification result; wherein the tongue image classification model is a pre-trained multi-task tongue image classification model, a multi-task network model, a cascade network model or a transformer network model; and the multi-task network comprises a tongue image feature classification layer, a dimension classification layer and a region classification layer.

[0015] Some embodiments of the present application classify each of the plurality of to-be-detected images by using a tongue image classification model to obtain tongue image classification results in multiple dimensions and multiple regions, which is more comprehensive and provides accurate support for subsequent tongue health state evaluation.

[0016] In some embodiments, the tongue health state is determined based on the dimension weight distribution table and the tongue image classification result of each of the plurality of to-be-detected images, which comprises: calculating a region probability of each of the plurality of dimensions in the plurality of regions based on the dimension weight distribution table; weighting and summing the region probability of each dimension and the corresponding category score and performing normalization processing to obtain a normalized score of each dimension; multiplying the normalized score and the dimension weight of each dimension to obtain a dimension score of each dimension; calculating the dimension score of each dimension to obtain the tongue score; and determining the tongue health grade corresponding to the tongue score.

[0017] Some embodiments of the present application evaluate the tongue health state by using a dimension weight distribution table and a tongue image classification result, realize quantitative analysis of the tongue image, track the change trend of the health state, and facilitate explanation and display to the user.

[0018] In some embodiments, after the tongue health state is determined, the method further comprises: generating a tongue health report of the target object; wherein the tongue health report comprises the dyeing moss category probability value, the tongue image classification result or the tongue health state.

[0019] Some embodiments of the present application can intuitively display the specific situation of the target object by generating a tongue health report.

[0020] In a second aspect, some embodiments of the present application provide a device for tongue image health state analysis, comprising: a cropping module configured to obtain a cropped image after cropping of an original tongue image of a target object; an enhancement module configured to perform image enhancement on the cropped image to obtain a plurality of detection images; a classification module configured to perform tongue image classification on each of the plurality of detection images to obtain a tongue image classification result of each detection image; wherein the tongue image classification result comprises a plurality of regions in a plurality of dimensions on the tongue image; and a health assessment module configured to determine a tongue image health state based on a dimension weight distribution table and the tongue image classification result of each detection image; wherein the tongue image health state is represented by a tongue image score and / or a tongue image health level; and the dimension weight distribution table comprises scores and weights of different regions of the tongue image under different dimensions.

[0021] In a third aspect, some embodiments of the present application provide a computer readable storage medium having stored thereon a computer program, which, when executed by a processor, can implement the method according to any one of the first aspect.

[0022] In a fourth aspect, some embodiments of the present application provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method according to any one of the first aspect.

[0023] In a fifth aspect, some embodiments of the present application provide a computer program product, comprising a computer program, wherein the computer program, when executed by a processor, can implement the method according to any one of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the following will briefly introduce the drawings needed to be used in some embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0025] Figure 1 A system diagram for tongue image health state analysis provided by some embodiments of the present application; Figure 2 One of the method flowcharts for tongue image health state analysis provided by some embodiments of the present application; Figure 3 The second method flowchart for tongue image health state analysis provided by some embodiments of the present application; Figure 4A device block diagram for tongue health state analysis is provided for some embodiments of the present application. Figure 5 An electronic device schematic diagram is provided for some embodiments of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in some embodiments of the present application will be described below with reference to the drawings in some embodiments of the present application.

[0027] It should be noted that similar reference numerals and letters indicate similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used for differentiation and cannot be understood as indicating or implying relative importance.

[0028] In the related art, one way is for a doctor to observe the tongue with naked eyes, judge the tongue color, tongue shape, moss quality, moss color, etc. according to experience, and then give a diagnosis conclusion; this method relies heavily on the doctor's experience and subjective judgment. Another way is to use a simple image processing method or a single deep learning model to analyze the tongue image; among them, traditional image processing techniques (such as color space conversion, threshold segmentation, edge detection, etc.) or neural networks (CNN, FNN, etc.) are used to analyze and classify tongue images (red tongue, cracked tongue, etc.).

[0029] However, the image processing method cannot accurately determine the tongue area; and some image processing methods only segment out the tongue to completely remove the background information, which means that the background information of the tongue is completely ignored. For example, the automatic white balance of the camera and the light environment will greatly affect the color performance. A tongue taken under yellow light and a tongue taken under white light will have a huge difference in RGB value. If a part of the background with known color (for example, an ideal standard color card, or even the user's facial skin) is retained, the algorithm can perform color correction and normalization on the entire picture based on this, so as to obtain a tongue color closer to the real one and not affected by the environment light. If the background is completely removed, it is not conducive to remove all background information blindly for tasks that focus on color recognition, and the final tongue image analysis result is also inaccurate. Moreover, the existing technology has limited dimensions for tongue image classification, and cannot fully evaluate the specific situation of the tongue image; if a single inference method is used, the performance of the same image under different light and angle will be quite different, and the lack of robustness leads to low accuracy of tongue image analysis. Moreover, the existing technology mostly only outputs simple classification results, lacks a quantitative evaluation system for tongue images, and cannot evaluate the tongue health state.

[0030] In view of this, some embodiments of the present application provide a tongue image health state analysis method, which can perform image enhancement on the cropped image after the original tongue image is cropped to obtain a plurality of to-be-detected images; then classify the plurality of to-be-detected images to obtain tongue image classification results in multiple dimensions and multiple regions; and finally determine the tongue image health state through a dimension weight distribution table and the tongue image classification results. The embodiments of the present application can realize comprehensive classification of tongue images from multiple dimensions and multiple regions, and realize accurate evaluation of tongue image health state, which is highly reliable and intuitive, and is conducive to health management and efficacy evaluation.

[0031] The overall structure of the tongue image health state analysis system provided by some embodiments of the present application will be described below with reference to the accompanying drawings. Figure 1 The overall structure of the tongue image health state analysis system provided by some embodiments of the present application will be described below with reference to the accompanying drawings.

[0032] As Figure 1 shown, some embodiments of the present application provide a system diagram of tongue image health state analysis. The tongue image health state analysis system can include a terminal 100 and a server 200. The terminal 100 can send an original tongue image of a target object to the server 200. The server 200 can obtain a plurality of to-be-detected images after cropping and enhancing the original tongue image; then perform tongue image classification on the plurality of to-be-detected images to obtain tongue image classification results. Finally, the tongue image health state is obtained by quantitatively analyzing the tongue image classification results according to a pre-set dimension weight distribution table.

[0033] In some embodiments of the present application, the terminal 100 can be a mobile terminal or a non-portable computer terminal, which is not specifically limited in the embodiments of the present application. In addition, various pre-trained models related to tongue image health state analysis are deployed in the server 200 to realize accurate and comprehensive classification and health evaluation of tongue images.

[0034] The implementation process of tongue image health state analysis performed by the server 200 provided by some embodiments of the present application will be described below with reference to the accompanying drawings. Figure 2 The implementation process of tongue image health state analysis performed by the server 200 provided by some embodiments of the present application will be described below with reference to the accompanying drawings.

[0035] Please refer to the accompanying drawings Figure 2 , Figure 2 A tongue image health state analysis method flowchart provided by some embodiments of the present application. The tongue image health state analysis method can include: S210, obtaining a cropped image after the original tongue image of the target object is cropped.

[0036] For example, in specific embodiments of the present application, the original tongue image is cropped according to a set proportion to obtain a cropped image. Alternatively, the original tongue image is cropped according to a suitable cropping frame to obtain a cropped image.

[0037] In some embodiments of the present application, S210 can include detecting the original tongue image to obtain a detection frame; wherein the detection frame includes a tongue region and a background region; expanding the detection frame to obtain an expanded frame; and cropping the original tongue image using the expanded frame to obtain the cropped image.

[0038] For example, in specific embodiments of the present application, a YOLOv5 target detection network is used to automatically locate the tongue in the original tongue image and expand the background region to output a detection frame. After the target detection network outputs the detection frame, the original detection frame size is expanded to ensure that the new detection frame contains the complete tongue and a certain proportion of the background region, thereby obtaining an expanded frame. Then, the original tongue image is cropped according to the expanded frame to obtain a cropped image.

[0039] In addition to using the YOLOv5 target detection network for detection frame detection, other network models can also be used, such as two-stage detection networks such as Faster R-CNN, Mask R-CNN, etc., or SSD (Single Shot MultiBox Detector), RetinaNet, DETR (Detection Transformer), etc.; as long as the detection frame and confidence of the tongue position can be output, the YOLOv5 target detection network can be replaced. The embodiments of the present application are not limited in this regard.

[0040] The expansion of the detection frame to obtain the expanded frame includes the following three implementation methods: 1) The detection frame is expanded according to a preset proportion to obtain the expanded frame.

[0041] For example, the width and height of the detection frame are calculated, and the original detection frame size is expanded by a certain expansion proportion (as a specific example of the preset proportion, for example, 25%) to obtain the expanded frame. The expansion proportion can be any value; the specific value can be set according to the actual application scenario, and the embodiments of the present application are not limited in this regard.

[0042] 2) The current size of the detection frame is obtained; and the expanded frame is obtained based on the expansion proportion corresponding to the current size.

[0043] For example, the current size is adaptively and dynamically adjusted according to the size of the detection frame to obtain the expanded frame. For example, for a small detection frame smaller than 200x200, the expansion proportion can be 30%; for a medium detection frame of 200x200-400x400, the expansion proportion can be 25%; and for a large detection frame larger than 400x400, the expansion proportion can be 20%. It can be understood that the correspondence between the size of the detection frame and the expansion proportion can be flexibly adjusted, and the embodiments of the present application are not limited in this regard.

[0044] 3) Obtain the confidence of the detection frame; based on the confidence corresponding to the expansion ratio, the expansion frame is obtained.

[0045] For example, the detection frame is adaptively dynamically adjusted according to the confidence. For example, when the confidence of the detection frame is greater than 0.9, the expansion ratio can be 20%; when the confidence of the detection frame is 0.5-0.9, the expansion ratio can be 25%; and when the confidence of the detection frame is less than 0.5, the expansion ratio can be 30%. It can be understood that the correspondence between the confidence of the detection frame and the expansion ratio can be flexibly adjusted, and the embodiments of the present application are not limited thereto.

[0046] In addition to the above several ways, other ways can be used to determine the detection frame, and the embodiments of the present application are not limited herein.

[0047] Since the existing tongue diagnosis system lacks a staining moss judgment function, in some embodiments of the present application, before S220 is performed, the tongue image health state analysis method can further include: inputting the cropped image into a pre-trained staining moss classification model to obtain a staining moss category probability value of the tongue image of the target object.

[0048] For example, in specific embodiments of the present application, a special staining moss classification module is introduced. The staining moss classification module uses a resnet18 network structure (as a specific example of the staining moss classification model) to extract features and classify the cropped image, and outputs the staining moss category probability value.

[0049] If the staining moss category probability value is greater than a set threshold value (for example, 0.86), it is determined to be staining moss, otherwise it is non-staining moss. The set threshold value can be any value directly between 0.1 and 1, and can be adaptively adjusted according to actual conditions. Alternatively, the optimal preset threshold value can be determined according to the staining moss data set statistics related to the tongue image. Alternatively, the optimal preset threshold value can be determined using an ROC curve.

[0050] Alternatively, a multi-threshold strategy can also be used to determine whether it is staining moss. For example, if the staining moss category probability value is greater than 0.9, it is determined to be staining moss; if the staining moss category probability value is in (0.7, 0.9], it can be staining moss, and a recheck is prompted in the subsequent tongue image health report; and if the staining moss category probability value is less than or equal to 0.7, it is determined to be non-staining moss.

[0051] It can be understood that the threshold values set above can be adaptively adjusted according to the requirements of sensitivity and specificity of the application scenario.

[0052] In addition, in addition to using the resnet18 network structure, deeper ResNet series such as ResNet50, ResNet101, EfficientNet series (B0-B7), MobileNet series (lightweight scheme), VGGNet, DenseNet, Vision Transformer (ViT), Swin Transformer, etc. can be used as long as the image classification network can realize binary classification.

[0053] It should be noted that the staining moss classification and the subsequent tongue feature classification are independently run, and the failure of the staining moss classification does not affect the tongue classification, and the two results can be integrated into the final report. This module can effectively distinguish between staining moss and pathological tongue moss, improve the reliability of diagnosis, and avoid misdiagnosis and over-treatment.

[0054] S220, image enhancement is performed on the cropped image to obtain a plurality of images to be detected.

[0055] For example, in the specific embodiments of the present application, a plurality of images to be detected are obtained by image enhancement for subsequent batch reasoning to improve the robustness of classification.

[0056] The implementation mode of image enhancement includes the following modes: 1) The cropped image is rotated at multiple angles and the brightness is adjusted, and a total of 4 images are generated (as a specific example of the plurality of images to be detected); the 4 images can include the cropped image and three images obtained by rotating 90°, 180° and 270°.

[0057] 2) The cropped image is flipped at multiple angles to obtain 4 images, which are the cropped image and three images obtained by horizontal flipping, vertical flipping and horizontal and vertical flipping.

[0058] 3) Only the cropped image and the two images after 180° rotation are used; and the images corresponding to the single-image mode and the fast mode of the cropped image.

[0059] 4) The cropped image is quality evaluated to determine whether to enhance, if it is a high-quality image (the quality score is greater than the quality threshold), it can not be enhanced; otherwise, image enhancement processing is required.

[0060] It can be understood that the mode and number of image enhancement can be adjusted according to the trade-off between accuracy and speed, which is not specifically limited in the embodiments of the present application.

[0061] The image enhancement mode can simulate different shooting angles and lighting conditions, and subsequently improve the robustness of the model to angle and lighting changes, and improve classification accuracy through subsequent multi-image reasoning, reduce the risk of misjudgment, and make the result more stable and reliable.

[0062] S230, tongue image classification is performed on each of the plurality of images to be detected, and the tongue image classification result of each detection image is obtained; wherein the tongue image classification result includes multiple regions in multiple dimensions on the tongue image.

[0063] For example, in the specific embodiments of the present application, batch inference is performed on the above obtained 4 images respectively, and the tongue image classification result of each image is obtained. Compared with the prior art which can only analyze a few dimensions and does not distinguish regions, the present application can realize tongue image classification results of 65 classifications of 13 dimensions and 5 regions (as one specific example of multiple dimensions and multiple regions). The specific content of the 65 classifications is shown in Table 1. The five regions of the following classifications conform to the theory of traditional Chinese medicine, such as tongue tip reflecting heart and lung, tongue root reflecting kidney, etc.

[0064] Table 1

[0065] The above 65 categories can be adjusted as needed (such as adding or deleting related regions, dimensions and categories), as long as the task design conforms to the classification of multiple regions, multiple dimensions and multiple categories. The specific region, dimension and category design can be adjusted according to the professional content of traditional Chinese medicine. The embodiments of the present application are not limited to this.

[0066] In some embodiments of the present application, S230 can include: inputting each image to be detected into a tongue image classification model, and outputting the tongue image classification result; wherein the tongue image classification model is a pre-trained multi-task tongue image classification model, a multi-task network model, a cascade network model or a transformer network model; the multi-task network includes a tongue image feature classification layer, a dimension classification layer and a region classification layer.

[0067] For example, in the specific embodiments of the present application, ResNet is used as the backbone network (as one specific example of a multi-task tongue image classification model), and the above obtained multiple images to be detected are processed to extract image features, while outputting 65 classification task results (as one specific example of tongue image classification results), which comprehensively analyzes the dimensions and covers all aspects of traditional Chinese tongue diagnosis. The multi-task tongue image classification model is provided with a shared feature extraction layer (i.e. a shared layer) and 65 independent classification heads (i.e. task heads); in addition, an attention module can be added between the shared layer and the task head to automatically learn the feature weight of different tasks.

[0068] In addition to using a multi-task tongue image classification model, a multi-task network model can also be used, which includes three layers, the first layer is a 2-branch classification of tongue body features and tongue fur features; the second layer is to further divide 13 dimensions in each branch; the third layer is to further divide 5 regions in each dimension, and the tongue image classification result is output in this way.

[0069] If a cascaded network model is used, it first analyzes the overall features of the image, then analyzes the regional features based on the overall features, and outputs the tongue image classification result. Alternatively, a Transformer structure (i.e., a transformer network model) can be used, specifically using a Transformer encoder to replace the CNN feature extraction layer and employing a multi-head attention mechanism to handle multiple tasks.

[0070] It is understood that tongue image classification models can be adaptively adjusted according to specific application scenarios, and the embodiments of this application are not specifically limited here.

[0071] S240, Based on the dimensional weight allocation table and the tongue image classification results of each detected image, determine the tongue image health status; wherein, the tongue image health status is represented by tongue image score and / or tongue image health level; the dimensional weight allocation table includes the score and weight of different regions of the tongue image under different dimensions.

[0072] For example, in a specific embodiment of this application, a dimension weight allocation table designed by a professional clinical TCM doctor is used. Combined with the tongue appearance classification results and the probability distribution of the 13 dimensions within the table, the scores of each dimension are accumulated to obtain a comprehensive percentage-based health score, and the level is confirmed, thereby quantitatively assessing the health status of the tongue appearance. The dimension weight allocation table includes a 13-dimensional weight allocation table and a linear score mapping algorithm table for each dimension. The 13-dimensional weight allocation table is shown in Table 2. Table 2

[0073] Taking tongue color as an example, the linear score mapping algorithm table is shown in Table 3 (i.e., an example of the mapping between 10 tongue color categories and the original score): Table 3

[0074] In addition to using a linear score mapping table as described above, nonlinear score mapping rules or multi-dimensional special rules can also be used. For example, if teeth marks are combined with tongue swelling / full mouth, a score of -0.2 can be assigned. The specific choice can be made according to the actual situation, and this application does not impose specific limitations on the embodiments herein.

[0075] In some embodiments of this application, S240 may include: calculating the regional probability of each of the plurality of dimensions under the plurality of regions based on the dimensional weight allocation table; weighting and summing the regional probability under each dimension with the corresponding category score and normalizing the sum to obtain a normalized score for each dimension; multiplying the normalized score with the dimensional weight of each dimension to obtain a dimensional score for each dimension; calculating the dimensional score for each dimension to obtain the tongue image score; and determining the tongue image health level corresponding to the tongue image score.

[0076] For example, in a specific embodiment of this application, the regional probabilities of five regions under each of the 13 dimensions in the tongue image classification result are fused. This fusion can be achieved by averaging the probability values ​​of the five regions, or by using a weighted average (e.g., a surface weight of 0.4 and other regions weights of 0.15). Alternatively, a voting mechanism can be used, where each of the five regions predicts its own category, and the probability corresponding to the category with the most occurrences is selected as the regional probability. Alternatively, the prediction result with the highest probability among the five regions can be used as the regional probability. Alternatively, the probability can be obtained by weighting the five regions according to their confidence levels, where regions with high confidence levels have larger weights. Specifically, the fusion method for the five regions can be selected based on the characteristics of different dimensions; this embodiment of the application does not impose specific limitations on this.

[0077] For example, taking tongue color as an example, after fusion, the resulting regional probability distribution is: {'pale red tongue': 0.6, 'pale white tongue': 0.3, 'red tongue': 0.1}.

[0078] After the above fusion is completed, for each dimension, the weighted sum is performed using the fused probability distribution: weighted_score = Σ(original score[category] × region probability[category]).

[0079] For example, as shown in Table 3, the original score is: weighted_score = 1.0×0.6 + 0.4×0.3 + 0.4×0.1 = 0.6 + 0.12 + 0.04 = 0.76.

[0080] Normalize the weighted_score using the following formula: normalized_score = (weighted_score + 0.3) / 1.3, which gives normalized_score = (0.76 + 0.3) / 1.3 = 0.815 (as a specific example of a normalized score).

[0081] Finally, the score for the tongue color dimension is obtained: dimension_score = 0.815×15 = 12.2 points (as a specific example of dimension score).

[0082] By following the above method, scores can be obtained for different dimensions. Finally, these scores are added together to obtain the scores for the tongue body and tongue coating respectively. For example, the tongue body scores 45 points and the tongue coating scores 55 points. The sum of the two scores gives the tongue appearance score.

[0083] By using a pre-constructed mapping table between tongue image scores and tongue image health levels, the corresponding tongue image health level can be obtained. For example, tongue image health levels can be divided into qualitative health levels such as healthy, sub-healthy, and mildly abnormal.

[0084] In some embodiments of this application, after S240 is executed, the method for analyzing the health status of the tongue image may further include: generating a tongue health report for the target object; wherein the tongue health report includes the probability value of the coating category, the tongue image classification result, or the tongue health status.

[0085] For example, in a specific embodiment of this application, a corresponding tongue health report can be generated based on one or more of the above-mentioned probability values ​​of tongue coating categories, tongue image classification results, and tongue health status. This tongue health report can group all categories according to tongue body characteristics and tongue coating characteristics, separate normal and abnormal dimensions, simplify the display of normal dimensions, provide detailed information on abnormal dimensions, and finally output in a structured JSON format. Example of output results: "Tongue characteristics": { "Normal dimensions": "The tongue is generally light red, of moderate thickness and size, without cracks or teeth marks"; "Abnormal Dimension": "Petechiae and Ecchymoses: Pinillitis on the tongue (Confidence: 0.75)" }, "Characteristics of tongue coating": { "Normal dimensions": "The tongue coating is thin, white, moist, evenly distributed, and has normal granules"; "Abnormal Dimension": ""}.

[0086] Besides JSON format, output can also be in XML format, Protobuf format, and more. It can also generate natural language reports, i.e., complete textual descriptions using natural language processing technology. Alternatively, it can generate visual reports, such as charts, heatmaps, or feature annotations on tongue images; or it can use a tiered output approach, for example, a simplified version outputting only the total score and health level; a standard version including the main dimensions; and a detailed version including all 65 dimensions. Understandably, the output format can be customized according to different application scenarios and user needs.

[0087] The following is in conjunction with the appendix Figure 3 The present application provides an exemplary description of the specific process of tongue health status analysis provided by some embodiments.

[0088] Please see the appendix Figure 3 , Figure 3 A flowchart illustrating a method for analyzing the health status of a tongue image, provided for some embodiments of this application.

[0089] The above process is illustrated below by example.

[0090] S310: Detect the original tongue image and obtain the detection box.

[0091] S320, expand the detection box to obtain an expanded box.

[0092] S330: Use an expanded frame to crop the original tongue image to obtain the cropped image.

[0093] S340, input the cropped image into the pre-trained tongue coating classification model to obtain the probability value of the tongue image of the target object belonging to the tongue coating category.

[0094] S350 performs image enhancement on the cropped image to obtain multiple images to be detected.

[0095] S360 inputs each image to be detected into the tongue image classification model and outputs the tongue image classification result.

[0096] S370 determines the health status of the tongue image based on the dimension weight allocation table and the tongue image classification results of each detected image.

[0097] S380, Generate a tongue health report for the target object.

[0098] The tongue health report includes the probability value of tongue coating category, tongue classification results, and tongue health status.

[0099] It should be noted that the specific implementation process of S310 to S380 can be referred to the method embodiment provided above. To avoid repetition, detailed descriptions are appropriately omitted here. Among them, the steps after S340 and S350 can be executed in parallel or in a certain order. This application embodiment does not make specific limitations here.

[0100] As can be seen from the above embodiments of this application, this application uses a YOLO object detection network + adaptive expansion algorithm for target determination. The YOLOv5 network, trained on a large amount of tongue image data, can accurately identify the tongue position with a detection accuracy >95%. The adaptive expansion algorithm expands the detection box according to a preset ratio, preserving complete tongue information while avoiding excessive background interference. Compared with existing background segmentation methods, this solution is more adaptable to different backgrounds, lighting, angles, and bright color paths in photography, and is fully automated, requiring no manual annotation, thus significantly improving efficiency.

[0101] Furthermore, this application is the first to achieve intelligent judgment of tongue coating staining, improving diagnostic reliability. It introduces a ResNet18 tongue coating staining binary classification model combined with adaptive threshold judgment. Existing tongue diagnosis systems generally lack tongue coating staining judgment, easily misclassifying stained tongue coating as pathological tongue coating. This application, through a specialized binary classification model, accurately identifies stained tongue appearance with an accuracy rate >92%. The adaptive threshold has been verified through extensive experiments, effectively balancing sensitivity and specificity, avoiding misdiagnosis caused by tongue coating staining, and improving diagnostic reliability and clinical application value.

[0102] This application utilizes a multi-task learning network and a five-region joint analysis design, employing 65 classification tasks to cover 13 core dimensions of TCM tongue diagnosis, making it more comprehensive than existing systems. The five-region analysis (tip, root, sides, center, and overall) aligns with TCM organ location theory (the tip reflects the heart and lungs: a red tip is common when heart fire is excessive; the root reflects the kidneys: a thick, greasy coating is common when kidney deficiency is present; the sides reflect the liver and gallbladder: a red side is common when liver stagnation transforms into fire; the center reflects the spleen and stomach: a thick coating is common when spleen deficiency is accompanied by dampness). Through a probabilistic fusion algorithm, information from the five regions is integrated to arrive at an overall assessment result, achieving refined and personalized health evaluation.

[0103] Furthermore, in the multi-angle rotation + batch inference + probability averaging algorithm, multi-angle rotation simulates different shooting angles, improving the model's robustness; inference is performed on four enhanced images separately, obtaining four sets of prediction results; through the probability averaging algorithm, the final probability of each category is the average of the probabilities from the four inferences. Averaging multiple measurements reduces random errors. Compared to single inference, the accuracy is improved by 5-10%, and the results are more stable and reliable. The improvement is particularly significant for tongue images with unclear boundaries and indistinct features. In short, the image enhancement + probability averaging method of this application exhibits high accuracy and stability.

[0104] Finally, this application improves the quantitative assessment system to facilitate health management. Specifically, the scoring algorithm incorporates expert knowledge for tiered assessment. Different dimensions are assigned different weights based on their importance in Traditional Chinese Medicine (TCM) theory (e.g., tongue color 15 points, coating thickness 12 points, etc.); non-linear scoring mapping reflects the severity of different categories (e.g., "pale red tongue" receives full marks, "purple spots" receive negative marks); special association rules (e.g., combined assessment of teeth marks and tongue swelling) incorporate TCM diagnostic thinking; the 100-point system is intuitive and easy to understand, and multiple health levels facilitate tiered management; detailed dimensional scoring supports targeted health interventions and efficacy tracking. Compared to existing systems that only output classification results, this solution provides users with more valuable information.

[0105] Please refer to Figure 4 , Figure 4 The diagram shows a block diagram of a tongue health status analysis apparatus provided in some embodiments of this application. It should be understood that this tongue health status analysis apparatus corresponds to the method embodiments described above and is capable of performing the various steps involved in the method embodiments. The specific functions of this tongue health status analysis apparatus can be found in the description above; detailed descriptions are omitted here to avoid repetition.

[0106] Figure 4 The device for analyzing the health status of tongue images includes at least one software functional module that can be stored in a memory or embedded in the device in the form of software or firmware. The device includes: a cropping module 410 for acquiring a cropped image of the original tongue image of the target object; an enhancement module 420 for enhancing the cropped image to acquire multiple images to be detected; a classification module 430 for classifying the tongue image of each of the multiple images to be detected, obtaining a tongue image classification result for each detected image; wherein the tongue image classification result includes multiple regions in multiple dimensions of the tongue image; and a health assessment module 440 for determining the health status of the tongue image based on a dimension weight allocation table and the tongue image classification result of each detected image; wherein the tongue image health status is represented by a tongue image score and / or a tongue image health level; the dimension weight allocation table includes the scores and weights of different regions of the tongue image in different dimensions.

[0107] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0108] Some embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can perform the operation of any of the methods corresponding to the methods provided in the above embodiments.

[0109] Some embodiments of this application also provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operation of any of the methods corresponding to the above embodiments provided in the above embodiments.

[0110] like Figure 5 As shown, some embodiments of this application provide an electronic device 500, which includes a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520. When the processor 520 reads the program from the memory 510 via a bus 530 and executes the program, it can implement the methods of any of the above embodiments.

[0111] Processor 520 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 520 can be a microprocessor.

[0112] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 520 of this disclosure embodiment can be used to execute the instructions in the memory 510 to implement the methods shown above. The memory 510 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0113] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0114] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0115] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for analyzing the health status of the tongue, characterized in that, include: Obtain the cropped image after cropping the original tongue image of the target object; Image enhancement is performed on the cropped image to obtain multiple images to be detected; Tongue image classification is performed on each of the multiple images to be detected to obtain the tongue image classification result for each detected image; wherein, the tongue image classification result includes multiple regions in multiple dimensions of the tongue image; Based on the dimensional weight allocation table and the tongue image classification results of each detected image, the health status of the tongue image is determined; wherein, the health status of the tongue image is represented by the tongue image score and / or tongue image health level; the dimensional weight allocation table includes the score and weight of different regions of the tongue image under different dimensions.

2. The method as described in claim 1, characterized in that, The cropped image obtained after cropping the original tongue image of the target object includes: The original tongue image is detected to obtain a detection box; wherein the detection box includes a tongue region and a background region; The detection box is expanded to obtain an expanded box; The original tongue image is cropped using the expanded frame to obtain the cropped image.

3. The method as described in claim 2, characterized in that, The step of expanding the detection box to obtain an expanded box includes: The detection frame is expanded according to a preset ratio to obtain the expanded frame; or... Obtain the current size of the detection frame; expand it based on the expansion ratio corresponding to the current size to obtain the expanded frame; or, Obtain the confidence level of the detection box; expand the detection box based on the expansion ratio corresponding to the confidence level to obtain the expanded box.

4. The method according to any one of claims 1-3, characterized in that, Before performing image enhancement on the cropped image to obtain multiple images to be detected, the method further includes: The cropped image is input into a pre-trained tongue coating classification model to obtain the probability value of the tongue image of the target object belonging to the tongue coating category.

5. The method according to any one of claims 1-3, characterized in that, The step of classifying the tongue image in each of the multiple images to be detected, and obtaining the tongue image classification result for each detected image, includes: Each image to be detected is input into the tongue image classification model, and the tongue image classification result is output; wherein, the tongue image classification model is a pre-trained multi-task tongue image classification model, multi-task network model, cascaded network model or transformer network model; the multi-task network includes a tongue image feature classification layer, a dimension classification layer and a region classification layer.

6. The method according to any one of claims 1-3, characterized in that, The determination of tongue health status based on the dimensional weight allocation table and the tongue image classification results for each detected image includes: Based on the dimension weight allocation table, calculate the regional probability of each of the multiple dimensions in the multiple regions; The region probability under each dimension is weighted and summed with the corresponding category score, and then normalized to obtain the normalized score for each dimension. The normalized score is multiplied by the dimensional weight of each dimension to obtain the dimensional score of each dimension. The score for each dimension is calculated to obtain the tongue image score; and the tongue image health level corresponding to the tongue image score is determined.

7. The method as described in claim 4, characterized in that, After determining the health status of the tongue image, the method further includes: Generate a tongue health report for the target object; wherein the tongue health report includes the probability value of the tongue coating category, the tongue classification result, or the tongue health status.

8. A device for analyzing the health status of the tongue, characterized in that, include: The cropping module is used to obtain the cropped image after cropping the original tongue image of the target object; The enhancement module is used to enhance the cropped image and obtain multiple images to be detected; The classification module is used to classify the tongue image in each of the multiple images to be detected, and obtain the tongue image classification result for each detected image; wherein, the tongue image classification result includes multiple regions in multiple dimensions of the tongue image; The health assessment module is used to determine the health status of the tongue image based on the dimensional weight allocation table and the tongue image classification results of each detected image; wherein, the health status of the tongue image is represented by tongue image score and / or tongue image health level; the dimensional weight allocation table includes the score and weight of different regions of the tongue image under different dimensions.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to perform the method as described in any one of claims 1-7.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the computer program is executed by the processor to perform the method as described in any one of claims 1-7.

Images