A multimodal tongue image analysis system

Abstract

The present invention provides a multimodal tongue image analysis system, relating to the field of image processing technology. The system comprises: a data acquisition unit for acquiring a user's tongue image and basic information; an image processing unit for extracting features from the tongue image to obtain multiple tongue image features; a tongue image recognition and classification model for classifying the tongue image features to obtain a tongue image feature map; an initial report generation unit for generating an initial tongue image analysis report based on the tongue image feature map and basic information; an interaction unit for determining the type of information required by the user based on the initial analysis report, thereby obtaining the user's supplementary information; and a final report generation unit for modifying the initial analysis report based on the user's supplementary information to obtain a final tongue image analysis report. By considering the user's basic and supplementary information, the present invention makes the analysis results more targeted, thereby improving the fit of the tongue image analysis results to the user.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multimodal tongue image analysis system. Background Art

[0002] With the development of computer vision and artificial intelligence technology, the application scope of image analysis technology continues to expand. In the field of user health consultation, especially for the analysis of user tongue images, image analysis technology usually analyzes the tongue image provided by the user according to its own parameters, and then feeds back an analysis result about the user's tongue to the user.

[0003] In related technologies, existing image analysis systems often ignore the individual differences of different users and are therefore unable to provide accurate analysis that meets the real needs of users, thereby affecting the practicality and reference value of the tongue image analysis results. Summary of the Invention

[0004] The problem solved by the present invention is how to improve the fit between tongue image analysis results and users.

[0005] To solve the above problems, the present invention provides a multimodal tongue image analysis system, comprising:

[0006] A data acquisition unit, configured to acquire a tongue image of a user and basic information of the user;

[0007] an image processing unit, configured to extract features from the tongue image to obtain a plurality of tongue image features of the tongue image; and classify the tongue image features using a tongue image recognition and classification model to obtain a tongue image feature map of the tongue image;

[0008] an initial report generating unit, configured to generate an initial analysis report of the tongue image based on the tongue image characteristic graph and the basic information;

[0009] an interaction unit, configured to determine, based on the initial analysis report, at least one type of information that the user needs to supplement, and then obtain the user's supplementary information based on the information type;

[0010] The final report generating unit is configured to amend the initial analysis report according to the supplementary information of the user to obtain a final analysis report of the tongue image.

[0011] Optionally, the data acquisition unit is specifically configured to:

[0012] When the tongue image of the user is obtained, the user's age, gender and related examination information are obtained through the information collection agent;

[0013] The age, gender and relevant examination information of the user are used as the basic information of the user.

[0014] Optionally, the image processing unit is specifically configured to:

[0015] performing standardization processing on the tongue image to obtain the standardized tongue image;

[0016] performing image segmentation on the tongue image after the normalization process, and separating the tongue area from the background in the tongue image;

[0017] Feature extraction is performed on the tongue region to obtain a plurality of tongue image features of the tongue image.

[0018] Optionally, the image processing unit is further configured to:

[0019] determining light intensity data of each pixel according to the brightness distribution of the tongue image;

[0020] generating a depth map of the tongue image according to the light intensity data, wherein the depth map includes a depth value of each pixel in the tongue image;

[0021] determining a boundary of the tongue area according to the depth value of each pixel;

[0022] The tongue region is separated from the background by the boundary.

[0023] Optionally, the image processing unit is further configured to:

[0024] performing feature extraction on the tongue region according to a preset color channel to obtain color features of the tongue region;

[0025] Identifying the edge of the tongue region using an edge detection method to obtain an edge pattern of the tongue region, and then identifying and extracting the edge pattern to obtain shape features of the tongue region;

[0026] determining a texture feature of the tongue region according to a pixel grayscale value of each pixel point in the tongue region;

[0027] Performing a depth analysis based on the texture features to obtain a tongue coating thickness feature of the tongue area;

[0028] The color feature, the shape feature, the texture feature and the tongue coating thickness feature are used as the tongue image features.

[0029] Optionally, the image processing unit is further configured to:

[0030] Performing grayscale transformation on each pixel point in the tongue area to obtain a grayscale value distribution of the tongue area;

[0031] determining a gray level co-occurrence matrix of the tongue region according to the gray value distribution of the tongue coating position in the tongue region;

[0032] extracting texture feature values ​​of the tongue region from the gray level co-occurrence matrix, wherein the texture feature values ​​include contrast, correlation, energy, and entropy of the tongue region;

[0033] The tongue coating thickness of the tongue area is obtained according to the contrast, correlation, energy and entropy of the tongue area, and features of the tongue coating thickness are extracted to obtain the tongue coating thickness features.

[0034] Optionally, the image processing unit is further configured to:

[0035] Dividing the tongue area into a plurality of sub-areas, and inputting all the tongue image features corresponding to each sub-area into the tongue image recognition and classification model;

[0036] Classifying the tongue image features of each sub-region using the tongue image recognition and classification model to obtain a classification result of the sub-region;

[0037] Marking is performed according to the classification results of all the sub-regions to obtain the tongue image feature map of the tongue region;

[0038] The tongue image feature map includes color distribution data, edge distribution data, texture structure data and tongue coating thickness data of the tongue area.

[0039] Optionally, the initial report generating unit is specifically configured to:

[0040] performing a similarity comparison between the color distribution data, the edge distribution data, the texture structure data, and the tongue coating thickness data in the tongue image characteristic graph and a plurality of corresponding preset data in a preset tongue image state knowledge base, and obtaining similarity values ​​between the color distribution data, the edge distribution data, the texture structure data, and the tongue coating thickness data and the plurality of corresponding preset data in the preset tongue image state knowledge base, respectively;

[0041] Determining the tongue image type and type degree of the user according to the preset data corresponding to the highest similarity values ​​of the color distribution data, the edge distribution data, the texture structure data, and the tongue coating thickness data, in combination with the basic information of the user;

[0042] generating a feature description of the tongue image type according to the tongue image type and the type degree corresponding to the tongue area;

[0043] The initial analysis report of the tongue image is obtained according to the characteristic description of the tongue image type.

[0044] Optionally, the interaction unit is specifically configured to:

[0045] Selecting the type of information that the user needs to supplement based on the relationship between the highest similarity value of the color distribution data, the edge distribution data, the texture structure data, and the tongue coating thickness data and a preset threshold;

[0046] Wherein, according to the data whose highest similarity is less than the preset threshold, determining the information type corresponding to the data;

[0047] A requirement description is generated according to the information type, and the supplementary information of the user is obtained through the requirement description.

[0048] Optionally, the interaction unit is specifically configured to:

[0049] Re-evaluate the tongue image type and the degree of the type in the initial analysis report based on the supplementary information;

[0050] Based on the evaluation results, determine whether there are any deviations in the initial analysis report;

[0051] When there is a deviation in the initial analysis report, the classification result of the tongue image type and the degree of the type are adjusted according to the evaluation result;

[0052] The adjusted classification result and the type degree are used to regenerate a characteristic description of the tongue image type, and the final analysis report is obtained based on the characteristic description.

[0053] The multimodal tongue image analysis system of the present invention first acquires a user's tongue image and basic information through a data acquisition unit. This serves as the foundation for the entire analysis system. By incorporating this basic information, the present invention accounts for individual differences, ensuring that the final analysis results are tailored to the user's specific circumstances. Feature extraction is then performed on the tongue image to obtain multiple tongue features. Furthermore, these features are classified using a tongue image recognition and classification model to generate a tongue image feature map. This allows for more precise location and identification of key features in the tongue image, providing a more accurate basis for subsequent analysis report generation. The tongue image feature map and the user's basic information are combined to generate an initial analysis report. For example, different users' tongue images correspond to different normal ranges. Incorporating information such as the user's age allows for a more accurate determination of whether the tongue image is abnormal. Based on the initial analysis report, the system then determines the type of user information that requires supplementation and obtains this supplementary information. This allows the system to dynamically mine user information that is helpful for accurate analysis based on the initial analysis results. For example, if the initial report identifies factors that may be associated with a user's tongue color, the system then determines the type of supplementary information based on these factors. The interaction unit then interacts with the user to obtain this supplementary information, thereby refining the analysis. Using supplementary information to amend the initial analysis report to obtain a final analysis report, incorporating supplementary information into the analysis results can best align with the user's actual situation, resolving the problem of inaccurate analysis results caused by prior art due to ignoring individual differences. By considering the user's basic information and supplementary information, the present invention fully accounts for individual differences, can more accurately analyze tongue images, making the analysis results more targeted, thereby better meeting the user's actual needs and improving the fit of tongue image analysis results to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of the structure of a multimodal tongue image analysis system according to one embodiment of the present invention;

[0055] Figure 2 4 is a flow chart of a multimodal tongue image analysis method in another embodiment of the present invention. DETAILED DESCRIPTION

[0056] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0057] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0058] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to"; the term "based on" means "based at least in part on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0059] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0060] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0061] In response to the problems existing in the above-mentioned related technologies, this embodiment provides a multimodal tongue image analysis system.

[0062] Combine Figure 1 and Figure 2 As shown, the multimodal tongue image analysis system provided by the embodiment of the present invention includes:

[0063] The data acquisition unit is used to acquire the tongue image of the user and the basic information of the user.

[0064] Specifically, the data acquisition unit is the fundamental starting point for the system's analysis and is responsible for collecting two key pieces of information. One is the user's tongue image, which serves as the direct basis for subsequent tongue analysis. This information is captured using a high-resolution camera or other image acquisition device to ensure clarity, appropriate lighting, and a complete display of the tongue's shape, color, and other detailed features. The other is the user's basic information, including but not limited to age, gender, physical condition, and lifestyle habits, such as smoking and drinking habits, and daily routines. This information provides important context for subsequent personalized analysis and lays the foundation for accurate tongue image analysis.

[0065] The image processing unit is used to extract features based on the tongue image to obtain multiple tongue image features of the tongue image; and classify the tongue image features through a tongue image recognition and classification model to obtain a tongue image feature map of the tongue image.

[0066] Specifically, the image processing unit utilizes image processing algorithms, such as deep learning-based convolutional neural networks (CNNs), to extract features from tongue images. This allows it to identify multiple tongue characteristics, including tongue color (e.g., pale red, pale white, bright red), tongue coating (e.g., thickness, color, and distribution), and tongue morphology (e.g., fatness, thinness, and the presence or absence of tooth marks). The extracted tongue features are then classified and integrated using a pre-trained tongue image recognition and classification model to generate a tongue image feature map. This detailed map clearly displays the locations and attributes of key tongue features, providing accurate image data support for generating the initial analysis report. In a preferred embodiment of the present invention, the image processing unit also stores each acquired tongue image and recognition result, using this data as algorithm training data to improve algorithm recognition accuracy. Through effective data management and utilization, the algorithm's recognition accuracy improves by approximately 5% annually, optimizing the efficiency of data storage and algorithm training.

[0067] An initial report generating unit is used to generate an initial analysis report of the tongue image based on the tongue image feature map in combination with the basic information.

[0068] Specifically, the initial report generation unit combines the tongue image feature graph with the user's basic information to generate an initial analysis report. Based on the various tongue image features marked in the tongue image feature graph and the user's basic information, the unit makes a preliminary assessment of the user's tongue image, taking into account factors such as the impact of age on tongue image (the tongue image of an elderly person may differ from that of a younger person); and the specific changes in the tongue coating of a smoker. For example, if the tongue image feature graph indicates a pale tongue color and the user's basic information indicates chronic fatigue, the initial report may preliminarily determine that the user has a tongue problem associated with chronic fatigue, with the pale tongue color corresponding to this.

[0069] The interaction unit is configured to determine, based on the initial analysis report, at least one type of information that the user needs to supplement, and then obtain the user's supplementary information based on the information type.

[0070] Specifically, the initial report is analyzed and interpreted to identify key points that may affect the accuracy of the analysis results and require further clarification, thereby determining at least one type of information that the user needs to supplement. For example, if the initial report indicates that the user has a thick and greasy tongue coating, but it is uncertain whether this is related to a recent greasy diet, the interactive unit will determine that dietary information is the type of information that needs to be supplemented. Through the system interface, questionnaires, dialog boxes, etc., the user will be asked specific questions such as the content of their recent diet and whether they have consumed spicy and greasy foods. This will obtain the user's supplementary information, allowing subsequent analysis to more comprehensively and accurately consider the impact of various factors on tongue appearance.

[0071] The final report generating unit is configured to amend the initial analysis report according to the supplementary information of the user to obtain a final analysis report of the tongue image.

[0072] Specifically, the final report generation unit uses the supplementary information provided by the user to revise and improve the initial analysis report, integrate the supplementary information into the previous analysis logic, and re-evaluate the tongue image characteristics. For example, if the supplementary information shows that the user has indeed consumed a lot of greasy food recently, the analysis of the characteristic of thick and greasy tongue coating may be adjusted accordingly, no longer overly leaning towards disease factors, but more considering the influence of diet. The corresponding content will be modified in the final report to provide a tongue image analysis report that is more in line with the user's actual situation, thereby effectively improving the fit between the tongue image analysis results and the user.

[0073] In a preferred embodiment of the present invention, the system serially connects the tongue analysis steps in a pipelined manner, achieving an efficient tongue analysis process through agent scheduling. The agent scheduling mechanism sequentially calls different agents in a preset order. First, the "Information Collection Agent" is called to obtain basic user information, such as age and gender. Next, the "Initial Analysis Agent" analyzes the user's tongue image to produce an initial analysis report. Next, the "Interaction Agent" determines the necessary supplementary information based on the initial analysis report and further refines this information through interaction with the user. Finally, the "Reporting Agent" outputs a complete analysis report. This pipelined agent scheduling method improves the efficiency and accuracy of tongue analysis. The interactive agent can be presented to the user as a 3D digital human. The 3D digital human, based on an optimized logical linkage life advice knowledge base, generates inquiries for supplementary information, which the user can respond to via voice or text.

[0074] The multimodal tongue image analysis system of the present invention first acquires a user's tongue image and basic information through a data acquisition unit. This serves as the foundation for the entire analysis system. By incorporating this basic information, the present invention accounts for individual differences, ensuring that the final analysis results are tailored to the user's specific circumstances. Feature extraction is then performed on the tongue image to obtain multiple tongue features. Furthermore, these features are classified using a tongue image recognition and classification model to generate a tongue image feature map. This allows for more precise location and identification of key features in the tongue image, providing a more accurate basis for subsequent analysis report generation. The tongue image feature map and the user's basic information are combined to generate an initial analysis report. For example, different users' tongue images correspond to different normal ranges. Incorporating information such as the user's age allows for a more accurate determination of whether the tongue image is abnormal. Based on the initial analysis report, the system then determines the type of user information that requires supplementation and obtains this supplementary information. This allows the system to dynamically mine user information that is helpful for accurate analysis based on the initial analysis results. For example, if the initial report identifies factors that may be associated with a user's tongue color, the system then determines the type of supplementary information based on these factors. The interaction unit then interacts with the user to obtain this supplementary information, thereby refining the analysis. Using supplementary information to amend the initial analysis report to obtain a final analysis report, incorporating supplementary information into the analysis results can best align with the user's actual situation, resolving the problem of inaccurate analysis results caused by prior art due to ignoring individual differences. By considering the user's basic information and supplementary information, the present invention fully accounts for individual differences, can more accurately analyze tongue images, making the analysis results more targeted, thereby better meeting the user's actual needs and improving the fit of tongue image analysis results to the user.

[0075] Optionally, the data acquisition unit is specifically configured to:

[0076] When the tongue image of the user is obtained, the user's age, gender and related examination information are obtained through the information collection agent;

[0077] The age, gender and relevant examination information of the user are used as the basic information of the user.

[0078] Specifically, after acquiring a tongue image, the data acquisition unit further uses an information collection agent to obtain the user's age, gender, and related medical examination information, and uses this information as the user's basic information. This information collection agent can be an intelligent module integrated into the system. It can automatically collect basic demographic information such as the user's age and gender, as well as medical examination information potentially relevant to tongue analysis, such as recent blood test results and other physical examination indicators, through user input, integration with other information systems, or access to external databases. This basic information provides important context for subsequent tongue analysis, helping the system more accurately understand the potential implications of tongue image characteristics.

[0079] In an embodiment of the present invention, the basic information of the user is automatically acquired and integrated by the information acquisition intelligent agent, which ensures that individual differences are fully considered in the tongue image analysis process, thereby improving the personalization and accuracy of the analysis results. For example, the tongue image characteristics of different age groups may be different, the tongue image of the elderly may be different from that of young people, and gender may also have a certain impact on the tongue image. In addition, relevant examination information can provide a multi-dimensional reference for tongue image analysis. For example, inflammatory indicators in blood tests may be associated with specific features in tongue images. Therefore, the integration of these basic information not only enriches the data basis for the analysis, but also makes the tongue image analysis results more targeted and practical, thereby improving the fit between the tongue image analysis results and the user.

[0080] Optionally, the image processing unit is specifically configured to:

[0081] performing standardization processing on the tongue image to obtain the standardized tongue image;

[0082] performing image segmentation on the tongue image after the normalization process, and separating the tongue area from the background in the tongue image;

[0083] Feature extraction is performed on the tongue region to obtain a plurality of tongue image features of the tongue image.

[0084] Specifically, the image processing unit first standardizes the tongue image, including operations such as image resizing, color correction, and brightness adjustment. This eliminates image differences caused by factors such as the camera and ambient lighting, ensuring consistency and accuracy in subsequent processing. Next, the standardized tongue image is segmented. Using segmentation techniques such as threshold-based, edge detection, or machine learning algorithms, the tongue region is precisely separated from the background, effectively removing background interference and focusing on the tongue itself, providing a pure target region for feature extraction. Finally, feature extraction is performed on the segmented tongue region. Using computer vision algorithms and deep learning models, key tongue features such as tongue color, tongue coating thickness, and tongue shape are identified and extracted, providing an important basis for subsequent tongue analysis.

[0085] In an embodiment of the present invention, the accuracy and reliability of tongue analysis can be significantly improved through the above-mentioned standardization, image segmentation, and feature extraction processes. Standardization ensures the consistency of image data, allowing images collected at different times and with different devices to be analyzed under unified standards; image segmentation accurately locates the tongue area, avoiding interference from background information on feature extraction, and improving the purity and representativeness of features; feature extraction can comprehensively and meticulously mine key information from tongue images, providing a solid data foundation for generating accurate analysis reports. This effectively solves the problem of insufficient analysis accuracy caused by image differences and background interference in existing technologies, thereby improving the performance and practicality of the entire tongue analysis system.

[0086] Optionally, the image processing unit is further configured to:

[0087] determining light intensity data of each pixel according to the brightness distribution of the tongue image;

[0088] generating a depth map of the tongue image according to the light intensity data, wherein the depth map includes a depth value of each pixel in the tongue image;

[0089] determining a boundary of the tongue area according to the depth value of each pixel;

[0090] The tongue region is separated from the background by the boundary.

[0091] Specifically, based on the brightness distribution of the tongue image, light intensity data for each pixel is determined. Brightness distribution essentially reflects the brightness and darkness of each region in the image. In a preferred embodiment of the present invention, the image can be grayscaled, converting a color image into a grayscale image for more intuitive analysis of brightness information. Each pixel in the RGB color image is converted to a corresponding grayscale value, which represents the light intensity data for that pixel. Furthermore, brightness distribution analysis may also involve calculating the image's histogram and counting the number of pixels at different grayscale levels to comprehensively understand the overall brightness and darkness distribution characteristics of the image. This brightness distribution is then determined based on the brightness and darkness distribution, providing the basic data for subsequent depth map generation. Next, a depth map of the tongue image is generated based on the light intensity data. Depth map generation can be inferred by leveraging the correlation between image brightness and depth. For example, based on a surface reflection imaging model, assuming the light source direction and reflection characteristics are known, depth information can be inferred by analyzing brightness differences between different regions. Alternatively, depth estimation methods based on image sequences can be used to construct a depth map using multiple tongue images from different perspectives or with different degrees of focus, utilizing parallax principles or depth cues. The generated depth map is stored in matrix form, with each element corresponding to the depth value of a pixel in the image. This represents the depth differences between different regions of the tongue image in three-dimensional space. After acquiring the depth map, the system determines the boundaries of the tongue region based on the depth value of each pixel. This is a critical step in the image processing process, as only by accurately determining the boundaries can the tongue region be separated from the complex oral background. During this process, the system analyzes changes in depth values, identifying the tongue boundaries by detecting sudden changes in depth values ​​or areas with large gradients. For example, there is often a certain depth difference between the tongue surface and other parts of the mouth (such as teeth and gums), which manifests as significant changes in depth values ​​in the depth map. By identifying these changes, the system can more accurately delineate the tongue's contours. After determining the boundaries, the system uses image segmentation techniques to completely separate the tongue region from the background.

[0092] In an embodiment of the present invention, depth information is introduced to solve the problem of low accuracy of traditional image segmentation methods when processing tongue images, thereby greatly improving the segmentation accuracy of the tongue area and the background. In particular, in areas such as the tongue root that are difficult to process using traditional methods, they can be accurately separated from the oral background, providing a purer and more accurate image basis for subsequent tongue feature extraction, thereby improving the accuracy and reliability of the entire tongue image analysis system.

[0093] Optionally, the image processing unit is further configured to:

[0094] performing feature extraction on the tongue region according to a preset color channel to obtain color features of the tongue region;

[0095] Identifying the edge of the tongue region using an edge detection method to obtain an edge pattern of the tongue region, and then identifying and extracting the edge pattern to obtain shape features of the tongue region;

[0096] determining a texture feature of the tongue region according to a pixel grayscale value of each pixel point in the tongue region;

[0097] Performing a depth analysis based on the texture features to obtain a tongue coating thickness feature of the tongue area;

[0098] The color feature, the shape feature, the texture feature and the tongue coating thickness feature are used as the tongue image features.

[0099] Specifically, when extracting features from the tongue region, the image processing unit operates according to a preset color channel to obtain the color features of the tongue region, thereby identifying color changes of the tongue, such as light red, pale white, and bright red. These color changes are important indicators for tongue image analysis. At the same time, the edges of the tongue region are identified through edge detection to obtain an edge graph of the tongue, which is then further analyzed for shape features, such as the fatness or thinness of the tongue and the presence or absence of tooth marks. These shape features are also important indicators for tongue image analysis. In addition, the texture features of the tongue are determined based on the pixel grayscale value of each pixel in the tongue region. Texture features can reflect the fineness and distribution of the tongue coating. Furthermore, in-depth analysis based on texture features can infer the thickness features of the tongue coating, which plays a key role in determining the state of body functions such as the digestive system. Ultimately, the color features, shape features, texture features, and tongue coating thickness features are integrated together to form a complete tongue image feature.

[0100] In this embodiment of the present invention, a multi-dimensional, comprehensive feature extraction method comprehensively and meticulously captures key information from tongue images, providing rich and accurate data support for subsequent tongue image analysis. Analysis of color channels can keenly capture subtle changes in tongue color; edge detection and shape analysis accurately identify the morphological characteristics of the tongue, providing an intuitive basis for tongue image analysis; and analysis of texture and tongue coating thickness delve into the details of the tongue image. Combining these features can lead to more accurate tongue image analysis results.

[0101] Optionally, the image processing unit is further configured to:

[0102] Performing grayscale transformation on each pixel point in the tongue area to obtain a grayscale value distribution of the tongue area;

[0103] determining a gray level co-occurrence matrix of the tongue region according to the gray value distribution of the tongue coating position in the tongue region;

[0104] extracting texture feature values ​​of the tongue region from the gray level co-occurrence matrix, wherein the texture feature values ​​include contrast, correlation, energy, and entropy of the tongue region;

[0105] The tongue coating thickness of the tongue area is obtained according to the contrast, correlation, energy and entropy of the tongue area, and features of the tongue coating thickness are extracted to obtain the tongue coating thickness features.

[0106] Specifically, the first step involves grayscale transformation of the tongue area. The purpose of grayscale transformation is to convert a color image into a grayscale image for subsequent processing. Grayscale transformation can be achieved using the following formula:

[0107] ;

[0108] in, is the intensity value of the red channel, usually ranging from 0 to 255, is the intensity value of the green channel, usually ranging from 0 to 255, The intensity value of the blue channel, usually ranging from 0 to 255. The grayscale value obtained for calculation usually ranges from 0 to 255. Through this formula, each color pixel can be converted into a grayscale value, thereby obtaining the grayscale value distribution of the tongue area. Next, the grayscale co-occurrence matrix (GLCM) is determined based on the grayscale value distribution of the tongue coating position in the tongue area. The grayscale co-occurrence matrix is ​​a commonly used texture analysis method that describes the texture characteristics of the image by statistically analyzing the spatial distribution of the grayscale values ​​of pixels in the image. The specific steps are as follows: Adjust the grayscale value distribution to a smaller range (for example, 0 to 255). Calculate the frequency of occurrence of grayscale value combinations between each pixel and its neighboring pixels. Four directions (0°, 45°, 90°, 135°) and different distances (usually 1) are usually considered. Normalize the co-occurrence matrix so that the sum of its elements is 1. Extract texture feature values ​​from the normalized grayscale co-occurrence matrix, including contrast, correlation, energy, and entropy. The calculation formulas for these feature values ​​are as follows:

[0109] Contrast ( ) is calculated as:

[0110] ;

[0111] in, The gray level of the current pixel, usually ranging from 0 to , The grayscale level of the pixel points adjacent to the current pixel point in a specific direction and distance, usually ranging from 0 to , is the position in the gray-level co-occurrence matrix value.

[0112] Correlation ( ) is calculated as:

[0113] ;

[0114] in, is the mean of the rows of the gray-level co-occurrence matrix, and the calculation formula is:

[0115] ;

[0116] is the mean of the column, and the calculation formula is:

[0117] ;

[0118] is the standard deviation of the rows of the gray-level co-occurrence matrix, and the calculation formula is:

[0119] ;

[0120] is the standard deviation of the column, calculated as:

[0121] ;

[0122] energy( ) is calculated as:

[0123] ;

[0124] in, is the position in the gray-level co-occurrence matrix The value of

[0125] entropy( ) is calculated as:

[0126] ;

[0127] in, is the position in the gray-level co-occurrence matrix The value of is the logarithmic function with base 2.

[0128] The extracted contrast, correlation, energy, and entropy are used to analyze the fineness and distribution of the tongue coating. For example, high contrast and high energy generally indicate a thicker tongue coating, while low contrast and low energy generally indicate a thinner tongue coating. Finally, based on the contrast and energy, the tongue coating thickness of the tongue region is determined, and features are extracted for the tongue coating thickness to generate the tongue coating thickness feature.

[0129] In an embodiment of the present invention, by combining grayscale transformation and grayscale co-occurrence matrix technology, the accuracy of tongue coating thickness feature extraction is significantly improved, and the texture feature values ​​extracted by the grayscale co-occurrence matrix can more accurately reflect the fineness and distribution of the tongue coating.

[0130] Optionally, the image processing unit is further configured to:

[0131] Dividing the tongue area into a plurality of sub-areas, and inputting all the tongue image features corresponding to each sub-area into the tongue image recognition and classification model;

[0132] Classifying the tongue image features of each sub-region using the tongue image recognition and classification model to obtain a classification result of the sub-region;

[0133] Marking is performed according to the classification results of all the sub-regions to obtain the tongue image feature map of the tongue region;

[0134] The tongue image feature map includes color distribution data, edge distribution data, texture structure data and tongue coating thickness data of the tongue area.

[0135] Specifically, the image processing unit divides the tongue region into multiple subregions, enabling more refined analysis of each tongue component and avoiding potential feature omissions or ambiguity associated with a global analysis. The dimensions of each region can be determined based on pre-set size data. Next, all tongue features corresponding to each subregion are input into a tongue image recognition and classification model. Exemplarily, this tongue image recognition and classification model is typically a deep learning model trained on a large amount of tongue image data, capable of identifying and distinguishing different tongue features. In a preferred embodiment of the present invention, the tongue image recognition and classification model can be a deep learning model based on ResNet50, a classic deep residual network with 50 layers. Its residual connections effectively address the vanishing gradient problem in deep networks, enabling it to learn deeper feature representations. In this embodiment, the ResNet50 model is trained on a large amount of image data labeled with tongue features. This data covers various tongue colors (e.g., light red, pale white, bright red), tongue coating thickness (e.g., thin, medium, thick), and tongue morphology (e.g., large, thin, with tooth marks). During training, the model learns how to extract key features from tongue images and map them to corresponding classification labels. When the tongue image features of each subregion are input into the ResNet50 model, the model progressively extracts features and performs classification through its convolutional, pooling, and fully connected layers. Ultimately, the ResNet50 model outputs a classification result for each subregion, such as "light red" tongue color or "medium" tongue coating thickness. These classification results are used to generate a tongue image feature map, providing the basis for the final tongue image analysis report. The tongue image recognition and classification model classifies the tongue image features of each subregion, generating a classification result for each subregion, such as "light red" tongue color or "medium" tongue coating thickness. Based on the classification results of all subregions, the tongue region is annotated. The annotation process graphically displays the classification results on the tongue image, ultimately forming a tongue image feature map. The tongue image feature map details data such as color distribution, edge distribution, texture structure, and tongue coating thickness, with each data item precisely mapped to a specific location and subregion on the tongue.

[0136] In an embodiment of the present invention, the processing method based on sub-region division and classification greatly improves the accuracy and meticulousness of tongue image analysis. Dividing the tongue area into multiple sub-regions for analysis one by one can capture the subtle feature differences of different parts of the tongue and avoid the information loss that may be caused by overall analysis. Each sub-region is classified by the tongue image recognition and classification model, and the tongue image feature map generated by combining multi-dimensional features such as color, edge, and texture is not only intuitive but also rich in information, providing detailed visual and data support for subsequent tongue image analysis.

[0137] Optionally, the initial report generating unit is specifically configured to:

[0138] performing a similarity comparison between the color distribution data, the edge distribution data, the texture structure data, and the tongue coating thickness data in the tongue image characteristic graph and a plurality of corresponding preset data in a preset tongue image state knowledge base, and obtaining similarity values ​​between the color distribution data, the edge distribution data, the texture structure data, and the tongue coating thickness data and the plurality of corresponding preset data in the preset tongue image state knowledge base, respectively;

[0139] Determining the tongue image type and type degree of the user according to the preset data corresponding to the highest similarity values ​​of the color distribution data, the edge distribution data, the texture structure data, and the tongue coating thickness data, in combination with the basic information of the user;

[0140] generating a feature description of the tongue image type according to the tongue image type and the type degree corresponding to the tongue area;

[0141] The initial analysis report of the tongue image is obtained according to the characteristic description of the tongue image type.

[0142] Specifically, after receiving the tongue image feature map, the initial report generation unit first compares the color distribution data, edge distribution data, texture structure data, and tongue coating thickness data in the tongue image feature map with the corresponding preset data in the preset tongue image state knowledge base (color distribution data, edge distribution data, texture structure data, and tongue coating thickness data, as well as the tongue image type and type degree corresponding to the above main sentence). The preset tongue image state knowledge base is a standard database containing a variety of common tongue image states, each of which has corresponding characteristic data such as color, edge, texture, and tongue coating thickness. The similarity comparison can be based on the Euclidean distance and cosine similarity between the feature map data and the knowledge base data to obtain the similarity value of each data type with the corresponding preset data in the knowledge base. Next, for the color distribution data, edge distribution data, texture structure data, and tongue coating thickness data, the preset data with the highest similarity is found. For example, the color distribution data has the highest similarity to the "light red" tongue condition in the knowledge base, the edge distribution data has the highest similarity to the "smooth edge" condition, the texture structure data has the highest similarity to the "fine texture" condition, and the tongue coating thickness data has the highest similarity to the "thin coating" condition. Then, based on basic information such as the user's age and gender, the user's tongue condition type and degree of type are determined. For example, if the user is older and the edge distribution data has the highest similarity to "tooth-marked edge," the tongue condition may be classified as "Qi deficiency." The degree of type is determined based on the similarity value, with higher similarity values ​​indicating a deeper degree of type. Based on the determined tongue condition type and degree, a feature description of the tongue condition is generated. For example, "Qi deficiency tongue condition: pale red tongue with tooth-marked edges, thin white tongue coating, and slightly enlarged tongue body, indicating possible moderate Qi deficiency." Finally, these feature descriptions are integrated into a complete initial analysis report.

[0143] In this embodiment of the present invention, by comparing the multi-dimensional data in the tongue image feature graph with a pre-set tongue image state knowledge base for similarity, it is possible to accurately identify and classify tongue image characteristics. Integrating this with the user's basic information further improves the accuracy and personalization of tongue image type determination. This data comparison and knowledge base matching approach makes the initial analysis report more scientific and reliable, effectively improving the intelligence and practicality of the entire tongue image analysis system and meeting the user's demand for high-precision tongue image analysis.

[0144] Optionally, the interaction unit is specifically configured to:

[0145] Selecting the type of information that the user needs to supplement based on the relationship between the highest similarity value of the color distribution data, the edge distribution data, the texture structure data, and the tongue coating thickness data and a preset threshold;

[0146] Wherein, according to the data whose highest similarity is less than the preset threshold, determining the information type corresponding to the data;

[0147] A requirement description is generated according to the information type, and the supplementary information of the user is obtained through the requirement description.

[0148] Specifically, after receiving the highest similarity values ​​for the color distribution data, edge distribution data, texture structure data, and tongue coating thickness data from the initial report generation unit, the interaction unit compares these highest similarity values ​​with a preset threshold. The preset threshold is a standard value derived from statistical analysis of a large amount of tongue image data and is used to determine whether the tongue image features sufficiently match the preset data in the knowledge base. If the highest similarity value for a particular data type falls below the preset threshold—for example, if the highest similarity value for the color distribution data is 0.6, while the preset threshold is 0.7—this indicates that the color distribution data does not match the preset data in the knowledge base sufficiently well, possibly due to other factors affecting the accuracy of the color distribution determination. The interaction unit then determines the information type corresponding to the color distribution data, namely, the type of user supplementary information related to color distribution, such as whether the user has recently taken medications that may alter tongue color or consumed foods that may have color. Next, the interaction unit generates a request description based on the determined information type, such as, "To more accurately analyze your tongue color, please provide information on whether you have recently taken medications that may alter tongue color or consumed foods that may have color." This request description is sent to the user through user-friendly interface prompts or push notifications to obtain the user's supplementary information. These supplementary information will help subsequent analysis to more fully understand the causes of abnormal tongue color distribution, thereby improving the accuracy of the analysis results. In a preferred embodiment of the present invention, information can be sent to the user through the 3D digital human, thereby completing the interaction with the user.

[0149] In this embodiment of the present invention, by comparing the highest similarity value with a preset threshold, it is possible to effectively identify tongue image feature types that require further information, thereby ensuring the reliability of the analysis results. This approach not only improves the intelligence level of the system but also enhances the user experience, allowing users to participate in the analysis process, provide key background information, generate analysis reports that are more consistent with the user's actual information, and improve the relevance of tongue image analysis results to the user.

[0150] Optionally, the interaction unit is specifically configured to:

[0151] Re-evaluate the tongue image type and the degree of the type in the initial analysis report based on the supplementary information;

[0152] Based on the evaluation results, determine whether there are any deviations in the initial analysis report;

[0153] When there is a deviation in the initial analysis report, the classification result of the tongue image type and the degree of the type are adjusted according to the evaluation result;

[0154] The adjusted classification result and the type degree are used to regenerate a characteristic description of the tongue image type, and the final analysis report is obtained based on the characteristic description.

[0155] Specifically, after obtaining the user's supplementary information, the interaction unit first re-evaluates the tongue image type and severity in the initial analysis report. This process compares the supplementary information with the characteristic descriptions in the initial analysis report. For example, if the supplementary information indicates that the user has recently taken medication that may change tongue color, and the initial analysis report lists the tongue image type as "Qi Deficiency" with a pale tongue color, the re-evaluation will consider the medication's impact on tongue color and determine the accuracy of the tongue image type and severity in the initial analysis report. Based on the results of the re-evaluation, the interaction unit determines whether there are any deviations from the initial analysis report. If a deviation exists, for example, if the supplementary information reveals that the tongue image type or severity assessment in the initial analysis report is inaccurate, the interaction unit will adjust the tongue image type classification and severity based on the supplementary information. For example, the tongue image type may be adjusted to "Drug-Influenced" or the severity may be adjusted to "Mild." The interaction unit then uses the adjusted classification and severity to regenerate the characteristic description of the tongue image type. For example, if the adjusted tongue image type is "drug-affected type", the feature description may be: "Due to recent medication, the tongue color has changed. The current tongue image may be affected by the medication and needs to be observed after stopping the medication." Finally, the final analysis report is generated based on the new feature description to more accurately reflect the user's tongue image.

[0156] In this embodiment of the present invention, by re-evaluating and adjusting the initial analysis report incorporating user supplementary information, the system can correct possible misjudgments and generate an analysis report that better reflects the user's actual situation. This not only increases user confidence in the analysis results but also makes tongue analysis more practical and valuable.

[0157] Although the present invention is disclosed as above, the scope of protection disclosed by the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A multimodal tongue image analysis system, characterized in that: include: A data acquisition unit, configured to acquire a tongue image of a user and basic information of the user; an image processing unit, configured to extract features from the tongue image to obtain a plurality of tongue image features of the tongue image; and classify the tongue image features using a tongue image recognition and classification model to obtain a tongue image feature map of the tongue image; an initial report generating unit, configured to generate an initial analysis report of the tongue image based on the tongue image characteristic graph in combination with the basic information; the initial report generating unit is specifically configured to: perform similarity comparison between the color distribution data, edge distribution data, texture structure data, and tongue coating thickness data in the tongue image characteristic graph and a plurality of corresponding preset data in a preset tongue image state knowledge base, and obtain similarity values ​​between the color distribution data, the edge distribution data, the texture structure data, and the tongue coating thickness data and the plurality of corresponding preset data in the preset tongue image state knowledge base, respectively; Determining the user's tongue image type and type degree based on the preset data corresponding to the highest similarity value of the color distribution data, the edge distribution data, the texture structure data, and the tongue coating thickness data, combined with the user's basic information; and generating a feature description of the tongue image type based on the tongue image type and type degree corresponding to the tongue area; Obtaining the initial analysis report of the tongue image according to the characteristic description of the tongue image type; An interactive unit is configured to determine, based on the initial analysis report, at least one type of information that the user needs to supplement, and then obtain the user's supplementary information based on the information type; the interactive unit is specifically configured to: select the type of information that the user needs to supplement based on a relationship between the maximum similarity value of the color distribution data, the edge distribution data, the texture structure data, and the tongue coating thickness data and a preset threshold; wherein, based on data whose maximum similarity is less than the preset threshold, determine the information type corresponding to the data; generate a requirement description based on the information type, and obtain the user's supplementary information based on the requirement description; The final report generating unit is configured to amend the initial analysis report according to the supplementary information of the user to obtain a final analysis report of the tongue image.

2. The multimodal tongue image analysis system according to claim 1, characterized in that: The data acquisition unit is specifically used to: When the tongue image of the user is obtained, the user's age, gender and related examination information are obtained through the information collection agent; The age, gender and relevant examination information of the user are used as the basic information of the user.

3. The multimodal tongue image analysis system according to claim 1, characterized in that: The image processing unit is specifically used for: performing standardization processing on the tongue image to obtain the standardized tongue image; performing image segmentation on the tongue image after the normalization process, and separating the tongue area from the background in the tongue image; Feature extraction is performed on the tongue region to obtain a plurality of tongue image features of the tongue image.

4. The multimodal tongue image analysis system according to claim 3, characterized in that: The image processing unit is further configured to: determining light intensity data of each pixel according to the brightness distribution of the tongue image; generating a depth map of the tongue image according to the light intensity data, wherein the depth map includes a depth value of each pixel in the tongue image; determining a boundary of the tongue area according to the depth value of each pixel; The tongue region is separated from the background by the boundary.

5. The multimodal tongue image analysis system according to claim 3, characterized in that: The image processing unit is further configured to: performing feature extraction on the tongue region according to a preset color channel to obtain color features of the tongue region; Identifying the edge of the tongue region using an edge detection method to obtain an edge pattern of the tongue region, and then identifying and extracting the edge pattern to obtain shape features of the tongue region; determining a texture feature of the tongue region according to a pixel grayscale value of each pixel point in the tongue region; Performing a depth analysis based on the texture features to obtain a tongue coating thickness feature of the tongue area; The color feature, the shape feature, the texture feature and the tongue coating thickness feature are used as the tongue image features.

6. The multimodal tongue image analysis system according to claim 5, characterized in that: The image processing unit is further configured to: Performing grayscale transformation on each pixel point in the tongue area to obtain a grayscale value distribution of the tongue area; determining a gray level co-occurrence matrix of the tongue region according to the gray value distribution of the tongue coating position in the tongue region; extracting texture feature values ​​of the tongue region from the gray level co-occurrence matrix, wherein the texture feature values ​​include contrast, correlation, energy, and entropy of the tongue region; The tongue coating thickness of the tongue area is obtained according to the contrast, correlation, energy and entropy of the tongue area, and features of the tongue coating thickness are extracted to obtain the tongue coating thickness features.

7. The multimodal tongue image analysis system according to claim 3, characterized in that: The image processing unit is further configured to: Dividing the tongue area into a plurality of sub-areas, and inputting all the tongue image features corresponding to each sub-area into the tongue image recognition and classification model; Classifying the tongue image features of each sub-region using the tongue image recognition and classification model to obtain a classification result of the sub-region; Marking is performed according to the classification results of all the sub-regions to obtain the tongue image feature map of the tongue region; The tongue image feature map includes the color distribution data, the edge distribution data, the texture structure data and the tongue coating thickness data of the tongue area.

8. The multimodal tongue image analysis system according to claim 1, characterized in that: The interaction unit is specifically used to: Re-evaluate the tongue image type and the degree of the type in the initial analysis report based on the supplementary information; Based on the evaluation results, determine whether there are any deviations in the initial analysis report; When there is a deviation in the initial analysis report, the classification result of the tongue image type and the degree of the type are adjusted according to the evaluation result; The adjusted classification result and the type degree are used to regenerate a characteristic description of the tongue image type, and the final analysis report is obtained based on the characteristic description.

Images