Tongue image processing method, device, equipment and storage medium

By using pre-trained tongue detection and segmentation models to detect and segment the tongue image, the problem of inaccurate tongue image segmentation in the prior art is solved, and fast and accurate tongue segmentation is achieved, which is suitable for clinical research Application.

CN118691608BActive Publication Date: 2024-12-03BEIJING ECON MEDICAL TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411162204.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-12-03
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

The existing tongue image segmentation technology is difficult to accurately and quickly segment the tongue from the tongue image, resulting in the processed image being unable to be practically applied to clinical research.

Method used

The tongue image is detected and segmented by pre-trained tongue detection model and segmentation model. The tongue image is positioned by the detection model and cropped image. Then the cropped image is input to the segmentation model for segmentation, and finally visualized according to the detection information. and splicing, explicitly mark the segmentation effect.

Benefits of technology

It realizes accurate and rapid segmentation of the tongue body from the tongue image, provides clear segmentation results, facilitates the application of clinical diagnosis and treatment, reduces the model reasoning time, and improves the accuracy and generalization ability of segmentation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118691608B_ABST
    Figure CN118691608B_ABST
Patent Text Reader

Abstract

The present invention relates to a tongue image processing method, device, equipment and storage medium. The tongue image processing method comprises: obtaining a tongue image to be processed, the tongue image comprising a tongue object; using the tongue image as an input of a pre-trained tongue detection model, detecting the tongue object in the tongue image, and outputting a detection result, the detection result comprising detection information of a detection area corresponding to the tongue object in the tongue image, a detection image obtained by explicitly marking the detection area in the tongue image, and a cropped image of the detection area; using the cropped image as an input of a pre-trained tongue segmentation model, segmenting the tongue object in the cropped image, and outputting a tongue segmentation map; splicing the visualized tongue segmentation map and the detection image according to the detection information, and displaying the spliced ​​detection image, so that the tongue can be accurately and quickly segmented from the tongue image, and subsequent diagnosis and treatment can be carried out in a timely manner based on the visualized tongue segmentation map.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a tongue image processing method, device, equipment and storage medium. Background Art

[0002] Tongue diagnosis in traditional Chinese medicine is one of the most important contents in visual diagnosis in traditional Chinese medicine. With the rapid development of the computer field, many research results on computer-assisted diagnosis and treatment have been continuously applied to the field of tongue diagnosis, and the collection of digital tongue images has become more and more common. However, in addition to the tongue body, the collected tongue images often contain a lot of background areas that are useless for tongue diagnosis. It is difficult for existing technologies to accurately and quickly segment the tongue body, resulting in the inability to actually apply the processed tongue body images to clinical research. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present disclosure provide a tongue image processing method, device, equipment and storage medium.

[0004] In a first aspect, the present disclosure provides a method for processing a tongue image, comprising:

[0005] Acquire a tongue image to be processed, wherein the tongue image includes a tongue body object;

[0006] Using the tongue image as an input of a pre-trained tongue detection model, detecting the tongue object in the tongue image, and outputting a detection result, wherein the detection result includes detection information of a detection area corresponding to the tongue object in the tongue image, a detection image obtained by explicitly marking the detection area in the tongue image, and a cropped image of the detection area;

[0007] Using the cropped image as input of a pre-trained tongue segmentation model, segmenting the tongue object in the cropped image, and outputting a tongue segmentation map;

[0008] According to the detection information, the tongue segmentation map after visualization processing and the detection image are spliced, and the spliced ​​detection image is displayed.

[0009] In a second aspect, an embodiment of the present disclosure provides a tongue image processing device, comprising:

[0010] An acquisition unit, used for acquiring a tongue image to be processed, wherein the tongue image includes a tongue body object;

[0011] a detection unit, configured to use the tongue image as an input of a pre-trained tongue detection model, detect the tongue object in the tongue image, and output a detection result, wherein the detection result includes detection information of a detection area corresponding to the tongue object in the tongue image, a detection image obtained by explicitly marking the detection area in the tongue image, and a cropped image of the detection area;

[0012] A segmentation unit, used for taking the cropped image as an input of a pre-trained tongue segmentation model, segmenting the tongue object in the cropped image, and outputting a tongue segmentation map;

[0013] The display unit is used to splice the tongue segmentation map after visualization and the detection image according to the detection information, and display the spliced ​​detection image.

[0014] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0015] Memory;

[0016] Processor; and

[0017] Computer programs;

[0018] The computer program is stored in the memory and is configured to be executed by the processor to implement the tongue image processing method as described above.

[0019] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the tongue image processing method as described above are implemented.

[0020] The tongue image processing method provided by the present disclosure includes: obtaining a tongue image to be processed, the tongue image including a tongue object; using the tongue image as an input of a pre-trained tongue detection model, detecting the tongue object in the tongue image, and outputting a detection result, the detection result including detection information of a detection area corresponding to the tongue object in the tongue image, a detection image obtained by explicitly marking the detection area in the tongue image, and a cropped image of the detection area; using the cropped image as an input of a pre-trained tongue segmentation model, segmenting the tongue object in the cropped image, and outputting a tongue segmentation map; splicing the visualized tongue segmentation map and the detection image according to the detection information, and displaying the spliced ​​detection image, so as to accurately and quickly segment the tongue from the tongue image, and explicitly mark the detection and segmentation effects on the original tongue image, so as to facilitate subsequent relevant personnel to timely conduct diagnosis and treatment based on the visualized effect map. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 A flowchart of a training method for a neural network model provided in an embodiment of the present disclosure;

[0024] Figure 2 A schematic diagram of a tongue sample image provided for an embodiment of the present disclosure;

[0025] Figure 3 A schematic diagram of detection and annotation information of a tongue sample image provided by an embodiment of the present disclosure;

[0026] Figure 4 A schematic diagram of a tongue sample image after data enhancement provided in an embodiment of the present disclosure;

[0027] Figure 5 A schematic diagram of the structure of a tongue detection model provided in an embodiment of the present disclosure;

[0028] Figure 6 A schematic diagram of the tongue detection model training loss provided in an embodiment of the present disclosure;

[0029] Figure 7 A flow chart of the tongue detection model processing provided in the embodiment of the present disclosure;

[0030] Figure 8 A schematic diagram of segmentation and annotation information of a cropped sample image provided by an embodiment of the present disclosure;

[0031] Fig. 9 A schematic diagram of the structure of the tongue segmentation model provided in an embodiment of the present disclosure;

[0032] Fig.10 A schematic diagram of the tongue segmentation model training loss provided in an embodiment of the present disclosure;

[0033] Fig.11 A schematic diagram of a process flow of a tongue image processing method provided in an embodiment of the present disclosure;

[0034] Fig.12 A flow chart of tongue segmentation model reasoning provided in an embodiment of the present disclosure;

[0035] Fig.13 A flow chart of the visualization process of tongue segmentation results provided in an embodiment of the present disclosure;

[0036] Fig.14A schematic flow chart of an interactive tongue image processing method based on a tongue detection model and a tongue segmentation model provided in an embodiment of the present disclosure;

[0037] Fig.15 A schematic diagram of the structure of a tongue image processing device provided in an embodiment of the present disclosure;

[0038] Fig.16 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0041] Before explaining this application in detail, it is first necessary to explain the terms involved:

[0042] Object Detection: A computer vision technology that aims to identify and locate target objects in images. It can not only identify the object categories in the image, but also give the bounding box of each object.

[0043] Segment Anything Model (SAM): An image segmentation model designed to solve a wide range of image segmentation tasks. It is highly versatile and has strong performance. It can segment the target area in the image with a small amount of labeled data or zero samples.

[0044] Training Epoch: refers to the process in which the entire training data set is completely fed into the neural network for forward propagation and back propagation. In each Epoch, the model traverses the entire training data set once and updates the model parameters (such as weights and biases) to minimize the loss function.

[0045] Loss function: used to quantify the gap between the model's predicted value and the true value. The output of the loss function is a scalar value. The larger the value, the greater the gap between the model's predicted result and the actual result, and the worse the model's performance.

[0046] Precision (P): An important indicator for measuring the performance of a classification model or a detection model, indicating the proportion of samples that are actually positive among all samples predicted to be positive.

[0047] Recall (R): An important indicator for evaluating model performance, which indicates the proportion of samples that are successfully predicted as positive among all samples that are actually positive.

[0048] Mean Average Precision (mAP): An important indicator for evaluating model performance. It combines precision and recall to comprehensively measure the application effect of the model under different thresholds. The higher the mAP, the better the model performance.

[0049] Introducing Masked Auto-encoders (MAE): Enhance the capabilities of auto-encoders by introducing masks on the input data.

[0050] Transformer: A deep learning model architecture for natural language processing (NLP) and other sequence learning tasks.

[0051] Vision Transformer (ViT): A visual attention model based on the Transformer architecture, used to process computer vision tasks. It uses the Transformer's self-attention mechanism to effectively capture global information in the image. It is mainly used to process large-size images and long-range dependency tasks.

[0052] Specifically, in TCM diagnosis and treatment, "inspection" is the first of the four diagnoses, which mainly includes facial diagnosis and tongue diagnosis. Computer vision technology can complete image information extraction and analysis, and judge the information. This technology is logically consistent with the "tongue diagnosis" in TCM inspection. However, due to factors such as the size, shape, color difference of the tongue, and the closeness of the color to the lips, the difficulty of tongue segmentation is greatly increased, resulting in jagged pixels on the edge of the tongue segmentation, incomplete segmentation of the main body, and lip mis-segmentation.

[0053] For the tongue segmentation method, there are currently four main methods, including threshold segmentation, template matching, region segmentation and neural network. The method process is as follows:

[0054] 1) Thresholding is used to segment an image into different regions. Its main purpose is to separate the image into foreground and background based on a certain threshold of pixel values. The main idea is to compare the pixel values ​​of the image with one or more thresholds and classify the pixels as foreground or background based on the comparison results. The simplest threshold segmentation method is global threshold segmentation, in which a single threshold is used to segment the image into two parts.

[0055] 2) Template Matching is a technique used in image processing and computer vision to find the location of a template image in a large image. The main idea is to use the sliding window technique to search for the area in the large image that is most similar to the template image, which is mainly accomplished by calculating the similarity between the window area and the template image.

[0056] 3) Region Segmentation is an image segmentation technique that divides an image into several non-overlapping regions, so that each region has consistency (such as similar color, brightness or texture) and there are obvious differences between regions. There are many methods for region segmentation, and each method has its own different ideas. For example, the idea of ​​Region Growing is to start from one or more seed points, add adjacent pixels to the current region according to the similarity criterion, and repeat this process until no new pixels can be added to the region. The idea of ​​Region Splitting and Merging is to recursively divide the image into smaller sub-regions until each sub-region meets certain similarity criteria, and finally try to merge similar small regions to obtain a larger segmented region.

[0057] 4) Neural Networks, which are composed of multiple layers, each containing multiple nodes. The basic neural network structure includes: input layer, hidden layers, and output layer. Each node is connected to the nodes of the previous layer through connection weights and biases. The calculated value is nonlinearly transformed through the activation function and passed to the next layer. Neural networks can solve the impact of environmental factors on tongue segmentation by learning features of different dimensions.

[0058] However, all four methods mentioned above have many problems, which makes it difficult to invest in clinical research related to tongue diagnosis. The first method, threshold segmentation, is very sensitive to lighting conditions. If there is uneven lighting or shadows in the image, the segmentation results may be seriously affected. And the changes in lighting in different areas will cause different parts of the same object to have different grayscale values, resulting in segmentation failure. And threshold segmentation is usually applied to grayscale images. For color images, they need to be converted to grayscale images first, which may lose some important color information, thus affecting the segmentation results. Since threshold segmentation assumes that the image can be segmented into foreground and background by a single threshold, it is often not true in images with complex and diverse backgrounds. If the grayscale value distributions of the background and foreground overlap, the segmentation effect will be poor. Finally, choosing a suitable threshold is also a key issue. Different thresholds will lead to completely different segmentation results. The method of automatically selecting thresholds performs well in some cases, but its applicability is relatively poor, especially when the grayscale distributions of the foreground and background are not clearly separated. Therefore, although the threshold segmentation method is simple and effective in some application scenarios, its shortcomings limit its application in complex image segmentation tasks. For images with complex backgrounds, lighting changes or multiple targets, other methods are often needed.

[0059] The second method, template matching, also has many disadvantages and limitations. First, it is insensitive to scale and rotation. If the size of the target object changes or rotates, the effect of template matching will be significantly reduced. Multiple matches are required for different scales and directions, which increases the computational complexity. Secondly, template matching is very sensitive to noise and occlusion in the image. Noise will change the grayscale value of the image, and occlusion will cause part of the template to not match the image, resulting in matching failure. In addition, the computational complexity of template matching is high, especially when searching in large images. It is necessary to calculate the similarity of each possible position of the image, and the amount of calculation is proportional to the size of the image and the template. Finally, template matching assumes that the shape of the template and the target area is fixed. If the target object is deformed (such as non-rigid deformation), the template matching method will not be able to effectively handle it. In addition, if the above problems occur, template matching may produce mismatches, and similar non-target areas may be mistakenly identified as target areas.

[0060] The third method, region segmentation method, usually needs to analyze and compare each pixel in the image, which has high computational complexity, especially when processing high-resolution images, the computational overhead will increase significantly. In addition, region segmentation methods (such as region growing) are very dependent on initial conditions (such as seed point selection). Different initial conditions may lead to different segmentation results, lacking stability and consistency. In addition, region segmentation is very sensitive to image noise. Noise will cause the attribute differences between adjacent pixels to become larger, thereby affecting the coherence of the region and may lead to over-segmentation or under-segmentation. In practical applications, many scenes contain overlapping objects. In this case, the region segmentation method may not be able to handle it correctly, mainly because the region segmentation assumes that the regions in the image are non-overlapping, and the region segmentation method usually needs to set some parameters (such as thresholds, similarity metrics, etc.), and the selection of these parameters has a significant impact on the segmentation results.

[0061] The fourth method, the neural network method, solves the influence of environmental factors on tongue segmentation by learning features of different dimensions, and integrates the trained algorithm through transfer learning, so that the accuracy of tongue segmentation is improved compared with the above three solutions. However, in the neural network models currently used, most of the tongue images are collected under standard environments. Using data sets collected under standard environments will lead to weak generalization and low robustness of the model. In addition, in the current tongue segmentation network model, the small model can quickly process tongue images, but the processed segmentation boundaries are not clear, mostly jagged, and even incomplete tongue segmentation may occur. Although the large model can achieve good segmentation effects, its parameters are very large, resulting in a low inference speed and long inference time of the model, and low efficiency in actual use.

[0062] In view of the above technical problems such as low accuracy and incomplete tongue segmentation, the present disclosure provides a tongue image processing method, which uses a tongue detection model and a tongue segmentation model to detect and segment the tongue image to obtain a complete and accurate tongue segmentation map. This is described in detail through one or more of the following embodiments.

[0063] Specifically, the tongue image processing method can be executed by a terminal or a server. Specifically, the terminal or the server can detect and segment the tongue object in the tongue image through a tongue detection model and a tongue segmentation model. The execution subject of the training method of the neural network model and the execution subject of the tongue image processing method can be the same or different.

[0064] For example, in one application scenario, the server trains a neural network model. The terminal obtains the trained neural network model from the server, and the neural network model refers to a tongue detection model and a tongue segmentation model. The terminal detects and segments the tongue object in the tongue image through the trained tongue detection model and the tongue segmentation model. The tongue image may be obtained by taking a picture by the terminal. Alternatively, the tongue image is obtained by the terminal from other devices. Alternatively, the tongue image is an image obtained by the terminal after performing image processing on a preset image, and the preset image may be obtained by taking a picture by the terminal, or the preset image may be obtained by the terminal from other devices. Here, no specific limitation is made to other devices.

[0065] It is understandable that the neural network model training method and tongue image processing method provided by the embodiments of the present disclosure are not limited to the above possible scenarios. Since the trained neural network model can be applied to the tongue image processing method, before introducing the tongue image processing method, the neural network model training method can be introduced below.

[0066] The following takes the server training neural network model as an example to introduce a neural network model training method, that is, the training process of the neural network model. It can be understood that the neural network model training method is also applicable to the scenario of terminal training neural network model.

[0067] Figure 1 A flowchart of a training method for a neural network model provided in an embodiment of the present disclosure is provided.

[0068] Among them, the training method of the neural network model specifically includes the following Figure 1 The following steps are shown:

[0069] S101: Acquire detection training data.

[0070] Among them, the detection training data includes multiple training data, each training data includes multiple tongue sample images, the multiple training data include first category training data, second category training data, third category training data and fourth category training data, the first category training data includes a series of tongue sample images collected by following the tongue object, the second category training data includes consistent tongue sample images collected under the same shooting conditions and ambient lighting, the third category training data includes diverse tongue sample images collected under different ambient lighting and / or shooting angles, and the fourth category training data includes tongue sample images collected in the target application scenario.

[0071] It is understandable that before model training, data collection operations need to be performed to produce high-quality training data sets. The training data sets include detection training data and segmentation training data. Among them, the detection training data used to train the tongue detection model includes multiple training data. The multiple training data are obtained based on different image sources, and each training data includes multiple tongue sample images. Specifically, the multiple training data include the first type of training data, the second type of training data, the third type of training data, and the fourth type of training data. The first type of training data includes a series of tongue sample images collected by following the tongue object. For example, the images in the first type of training data come from a traditional Chinese medicine intelligent robot. The traditional Chinese medicine intelligent robot can be equipped with a 10-inch large screen and a 2000mAH large-capacity battery. It can intelligently follow the tongue and face when collecting images to ensure that the collected images are of high quality and clear details. In addition, in order to reduce the impact of ambient light on the collected images, the traditional Chinese medicine intelligent robot can also use a standard color card to restore the most realistic image. The model and type of the traditional Chinese medicine intelligent robot used are not limited. The second type of training data includes consistent images collected under the same shooting conditions and ambient lighting. For example, the second type of training data comes from professional relevant personnel and can be collected by relevant personnel through strict control of shooting conditions and ambient lighting to ensure the clarity and consistency of each image. The third type of training data includes diverse images collected under different ambient lighting and / or shooting angles. For example, the third type of training data comes from public data on the Internet. Public data on the Internet provides a large number of images under different lighting, angles and environments, which helps to enrich the diversity and generalization ability of the data set. The fourth type of training data includes images collected in the target application scenario. For example, the fourth type of training data comes from the traditional Chinese medicine tongue diagnosis scenario. Images can be collected in actual tongue diagnosis scenarios to improve the segmentation accuracy of the model in the tongue diagnosis scenario. Specifically, the collection details of various training data collected from multiple sources are shown in Table 1:

[0072] Table 1:

[0073]

[0074] It can be understood that Table 1 shows a total of 1789 tongue sample images, and the various training data may include multi-scale images such as 3072x4096, 3072x2548, 1536x2048, 1080x100, 580x650 and 360x360, and other possible scale images are not limited.

[0075] S102: Train the pre-built tongue detection model according to the detection training data to obtain a trained tongue detection model.

[0076] It is understandable that, based on the above S101, a pre-built tongue detection model is obtained. The tongue detection model can be a pre-trained detection model. The pre-trained detection model is trained according to the detection training data. For the tongue detection task, the model needs to identify the tongue features such as the shape, color, and texture of the tongue, so as to accurately locate and detect the tongue in the tongue sample image, and obtain a tongue detection model that can accurately identify and locate the tongue. It is understandable that the tongue sample image includes a tongue object and a background object. The background object can be a part of a human face and other collected backgrounds. The tongue detection model detects the tongue object from the tongue sample image to distinguish the tongue object from the background object.

[0077] Optionally, the pre-built tongue detection model is trained according to the detection training data in S102 to obtain a trained tongue detection model, which can be specifically implemented by the following steps:

[0078] Preprocess multiple tongue sample images included in the detection training data to determine the color, brightness and color space of the input data corresponding to the tongue detection model and the tongue segmentation model; mark the detection area corresponding to the tongue object in each tongue sample image to obtain the detection annotation information of each tongue sample image; perform data enhancement processing on the tongue sample image and / or the detection annotation information to expand the detection training data of the tongue detection model; predict the detection area of ​​the tongue object in the tongue sample image through the tongue detection model to obtain the detection sample information; calculate the first loss according to the coordinate information of each pixel point in the detection area included in the detection annotation information and the detection sample information, and adjust the parameters of the tongue detection model according to the first loss.

[0079] It is understandable that the tongue sample images collected in the detection training data need to be preprocessed and applied to subsequent data annotation and model training. Specifically, the image preprocessing operation mainly includes the following parts: First, using a standard color card to correct the tongue sample image can effectively reduce the impact of ambient light on the image color and brightness, and also help ensure that the tongue sample images under different acquisition conditions have consistent color features and brightness features, thereby improving the training effect of the model. Secondly, processing the tongue sample image through noise removal, color correction, and standardized image brightness can significantly improve the image quality, help ensure the consistency of the model input data, and reduce errors and noise interference caused by differences in image quality. Finally, through color space conversion and other technologies, the tongue sample image is converted to a color space that is more suitable for target detection and segmentation algorithms, which helps to highlight the characteristics of the tongue and enables the algorithm to more accurately identify and segment the tongue area.

[0080] For example, see Figure 2 , Figure 2The schematic diagram of the tongue sample image provided in the embodiment of the present disclosure specifically represents the contrast effect of the tongue sample image before and after preprocessing.

[0081] It is understandable that after completing the preprocessing of the tongue sample image, data annotation is performed on each tongue sample image. Data annotation provides necessary supervision information for the deep learning model, which can significantly improve the training effect of the model, and the accuracy of data annotation is crucial for training the tongue detection model. High-quality annotated data can significantly improve the accuracy and reliability of the model. Specifically, LabelImg can be used for data annotation. LabelImg is an image annotation tool used to manually annotate image data. It can annotate various types of object areas such as rectangular borders and polygons to generate training data for computer vision tasks. It is suitable for various tasks such as object detection and image segmentation.

[0082] For example, see Figure 3 , Figure 3 A schematic diagram of the detection and annotation information of the tongue sample image provided by the embodiment of the present disclosure, such as Figure 3 As shown in FIG. 1 , the tongue object in the preprocessed tongue sample image is annotated by a detection frame, wherein the detection annotation information includes the coordinate information of the target pixel point in the detection frame in the tongue sample image and the width and height of the detection frame, such as Figure 3 As shown, the target pixel point can be any pixel point of the four vertices of the detection box in the tongue sample image, or it can also be a pixel point of the center point of the detection box in the tongue sample image. The area selected in the detection box can be understood as the detection area, and the detection box selects the complete tongue object in the tongue sample image.

[0083] It is understandable that after completing the data annotation of the tongue sample image, data enhancement processing is performed on the tongue sample image and / or the detection annotation information to expand the detection training data of the tongue detection model. At this time, the detection training data includes the tongue sample image and the detection annotation data of the tongue sample image. Specifically, data enhancement can significantly improve the training effect, generalization ability and practical application performance of the model. Specifically, the corresponding algorithm can be written to complete random rotation, flipping, scaling and other operations on the tongue sample image and the detection annotation information to expand the data set so that the model can better adapt to the tongue sample images under different lighting and angle conditions. Based on the detection training data shown in Table 1 above, the detection training data after data enhancement is shown in Table 2:

[0084] Table 2:

[0085]

[0086] Understandably, during the data augmentation process, some images will fail to be scaled or cropped because their pixel values ​​are too small or the tongue accounts for too large a proportion, resulting in the inability to perform augmentation. This may cause the number of augmented samples after random scaling and random cropping to be inconsistent with the number of input samples. As shown in Table 2, the total number of samples after data augmentation is 10,359.

[0087] For example, see Figure 4 , Figure 4 A schematic diagram of a tongue sample image after data enhancement provided in an embodiment of the present disclosure, including an original tongue sample image (original image) and an image after data enhancement processing such as random rotation, random flipping, random scaling, random cropping and random brightness.

[0088] It can be understood that after completing the data enhancement, all the enhanced data and the original sample data are combined into the tongue detection data set of the tongue detection model. Subsequently, the tongue detection data set is divided into a training set, a validation set, and a test set according to a preset ratio to produce a data set that supports the training and testing of the tongue detection model. The specific data information included in the tongue detection data set is shown in Table 3:

[0089] Table 3:

[0090]

[0091] It can be understood that Table 3 randomly divides the data set into training set, validation set and test set in a ratio of 8:1:1 based on the data shown in Table 2 above.

[0092] It is understandable that the tongue detection model is trained through the tongue detection dataset. The model training process is the process of establishing the mapping relationship between input data and output results. Through a large amount of labeled training data, the model can learn how to convert input images into accurate detection results. Among them, the tongue detection model is composed of a backbone network, a fully connected layer, and an output layer, such as Figure 5 As shown, after retraining and fine-tuning the model, the optimal weight of the model can be obtained to detect the acquired new tongue image.

[0093] For example, see Figure 5 , Figure 5A structural schematic diagram of a tongue detection model provided for an embodiment of the present disclosure, wherein the tongue detection model includes multiple backbone networks, multiple fully connected layers, and an output layer. The multiple backbone networks are connected in sequence, and each backbone network has a corresponding fully connected layer. The input tongue sample image is sequentially extracted through the backbone network to extract tongue features such as tongue shape, color, and texture. The fully connected layer then combines the extracted features as input data for the output layer. The output layer outputs the type of the tongue sample image and the detection sample information of the tongue object in the tongue sample image, wherein the detection sample information specifically includes pixel coordinates (x, y) and the width and height (w, h) of the detection frame.

[0094] It is understandable that after obtaining the detection sample information of the tongue object in each tongue sample image predicted by the tongue detection model, the first loss is calculated based on the detection annotation information and the coordinate information of each pixel point in the detection frame involved in the detection sample information, and the parameters of the tongue detection model are adjusted according to the first loss. It is understandable that each tongue sample image has the detection annotation information as the real label and the detection sample information output by the model as the predicted label. In order to cope with the particularity of the tongue detection task, targeted model adjustments and fine-tuning of different parameters of the model can be performed to ensure that the tongue position can be reliably identified under different lighting intensities, angles and tongue appearance conditions.

[0095] For example, see Figure 6 , Figure 6 A schematic diagram of the tongue detection model training loss provided in the embodiment of the present disclosure is specifically the loss (Loss) on the training set and the validation set obtained in 200 rounds of training, which is recorded as the first loss. The first loss can be a cross-entropy loss (Cross-Entropy Loss), and the calculation formula is shown in formula (1):

[0096] Formula (1)

[0097] In the formula, It is the unique encoding of the true label (0 or 1 in the case of binary classification), that is, the label of the pixel in the detection annotation information, which is used to identify whether the pixel should be in the detection box. is the category probability distribution predicted by the model, that is, the label of the pixel in the detection sample information, and is also used to identify whether the pixel should be in the detection box. n is the number of samples, that is, the number of tongue sample images.

[0098] S103. According to the detection sample information of each tongue sample image output by the tongue detection model, the detection area in each tongue sample image is cropped out to obtain segmentation training data.

[0099] It can be understood that, based on the above S102, the tongue detection model obtained by training is used to output the detection sample information of the detection frame of the tongue object in the tongue sample image, and the tongue sample image is cropped and saved according to the detection frame to obtain a cropped sample image (called a tongue area map). The cropped sample image is then used to make a subsequent segmentation algorithm data set. There are two purposes for cropping the tongue sample image: one is to reduce the complexity of the image background so that the tongue segmentation model can have better segmentation performance; the other is to reduce the image resolution input to the tongue segmentation model, speed up model reasoning, and reduce output time.

[0100] For example, see Figure 7 , Figure 7 The tongue detection model processing flow chart provided for the embodiment of the present disclosure can be understood as the input and output process of the tongue detection model. Specifically, the tongue sample image is input into the tongue detection model, and the tongue detection model processes the tongue sample image to output the original-size detection sample image with the detection frame explicitly marked and the cropped sample image with the detection frame area (detection area) cropped out, wherein the model processing includes operations such as feature extraction, recognition, classification, and cropping.

[0101] S104: Train the pre-built tongue segmentation model according to the segmentation training data to obtain a trained tongue segmentation model.

[0102] It is understandable that, based on the above S103, a pre-trained tongue segmentation model is obtained, and the tongue segmentation model can be a SAM model. The segmentation training data is used as input, and the pre-trained tongue segmentation model is fine-tuned so that it can adapt to the tongue segmentation task. It is understandable that the SAM model has been pre-trained and has a wide range of segmentation capabilities. In order to achieve optimal performance in specific tasks (such as tongue segmentation tasks), it needs to be fine-tuned. During the fine-tuning process, the parameters of the model will be optimized according to the data of the specific task, so that the model can better identify and segment the tongue object, and improve the accuracy of the segmentation and the ability to retain details.

[0103] Optionally, the pre-built tongue segmentation model is trained according to the segmentation training data in S104 to obtain a trained tongue segmentation model, which can be specifically implemented by the following steps:

[0104] Pixel annotation is performed on the tongue object in each cropped sample image to obtain segmentation annotation information of each cropped sample image, wherein the multiple pixel points annotated in the segmentation annotation information represent the boundary and shape features of the tongue object; prompt information is set that the segmentation object is the tongue object, and the tongue object in the cropped sample image is segmented out based on the prompt information through a tongue segmentation model to obtain a segmented sample image; a second loss is calculated according to the pixel points in the segmented sample image and the segmentation annotation information by combining block loss and cross entropy loss, and the parameters of the tongue segmentation model are adjusted according to the second loss.

[0105] The segmentation training data includes a plurality of cropped sample images.

[0106] It is understandable that the cropped sample image is used as the training data of the tongue segmentation model, and the LabelImg software is used to annotate it to obtain the segmentation annotation information. The segmentation annotation information is annotated on the pixels, and multiple pixels represent the boundary and shape of the tongue. The segmentation annotation information provides the model with detailed pixel-level annotation information, enabling the model to learn the precise boundary and shape characteristics of the tongue, which is crucial for training a high-precision tongue segmentation model.

[0107] For example, see Figure 8 , Figure 8 This is a schematic diagram of the segmentation and annotation information of the cropped sample image provided in an embodiment of the present disclosure. The area composed of multiple pixel points annotated by the segmentation and annotation information can be understood as the tongue area, and the boundary and shape of the tongue object can be clearly defined.

[0108] It can be understood that the tongue segmentation dataset is composed of the cropped sample images and the segmentation annotation information of the cropped sample images. As the training dataset of the tongue segmentation model, the dataset can also be divided into a training set, a validation set and a test set according to a preset ratio to produce a dataset that supports the training and testing of the tongue segmentation model, wherein the preset ratio can be 8:1:1. The specific information of the tongue segmentation dataset is shown in Table 4:

[0109] Table 4:

[0110]

[0111] Understandably, the tongue segmentation model is fine-tuned using the prepared tongue segmentation dataset. The tongue segmentation model includes an image encoder, multiple image embeddings, a mask decoder, and a prompt encoder, such as Fig. 9As shown. Among them, the image encoder is used to encode the cropped sample image into a vector form to obtain image data, the image embedding is used to convert the image data into a continuous, low-dimensional vector representation, the prompt encoder is used to encode the prompt information into a vector form, the prompt information is to prompt the model that its segmentation object is the tongue object, and the mask decoder is used to fuse the two features output by the prompt encoder and the image embedding and decode the final segmentation mask image, which refers to the segmentation sample image. Among them, the image encoder uses the visual attention model (VisionTransformer, ViT) pre-trained by the autoencoder model (Masked Auto-encoders, MAE) as the image encoder, which can adapt to high-resolution input.

[0112] For example, see Fig. 9 , Fig. 9 A structural schematic diagram of a tongue segmentation model provided in an embodiment of the present disclosure, wherein a cropped sample image is used as an input image of the tongue segmentation model, the input image is encoded into a vector form by an image encoder, the output of the image encoder is dimensionally transformed by multiple image embeddings, and the outputs of the multiple image embeddings are integrated by multiple convolutions, the prompt information is encoded into a vector form by a prompt encoder, the integrated features are fused with the features output by the prompt encoder by a mask decoder, and a segmented sample image is decoded, wherein the segmented sample image includes the original pixel points of the tongue object in the tongue sample image and the background pixel points whose pixel values ​​are preset values, that is, the tongue object is segmented from the cropped sample object, and the original pixel points of the tongue object are not changed, and the pixel values ​​of the remaining background objects except the tongue object are set to the same preset value.

[0113] It is understandable that after completing the fine-tuning training of the tongue segmentation model, the effect of the tongue segmentation model in tongue segmentation was evaluated in detail. The training loss is as follows: Fig.10 As shown, Fig.10 The diagram of the training loss of the tongue segmentation model provided in the embodiment of the present disclosure is as follows. The loss adopts a combination of Dice Loss (block loss) and Cross-entropy Loss (cross entropy loss), and the second loss is calculated according to the pixel points in the segmentation sample image and the segmentation annotation information. Among them, Dice Loss is usually used for image segmentation tasks, especially for the segmentation of unbalanced categories. It is based on the Dice coefficient to measure the similarity between the segmentation result predicted by the model and the actual segmentation result. The calculation formula is shown in formula (2):

[0114] Formula (2)

[0115] in: It is the binary version of the true label, that is, the pixels in the segmentation annotation information; It is the binary version of the model prediction, that is, the pixels in the segmented sample image. For example, the pixels marked as 0 represent background objects, and the pixels marked as 1 represent tongue objects. is a small constant that prevents the denominator from reaching zero.

[0116] It can be understood that when the tongue segmentation model performs the image segmentation task, the Dice Loss and Cross-Entropy Loss are combined to consider both the pixel-level similarity and the overall classification accuracy. The weighted average can be used to combine the two loss functions. The calculation formula is shown in formula (3):

[0117] Formula (3)

[0118] In the formula, is a weight parameter used to balance the relative importance of the two loss functions.

[0119] It can be understood that by comparing the segmented sample image with the true segmentation result (segmentation annotation information), the algorithm's segmentation accuracy under different tongue features and background conditions is verified, and the mean average precision (mAP), precision, recall, and F1 are used as test indicators to ensure that the tongue segmentation model can achieve accurate segmentation of the tongue under various conditions. The segmentation indicators for various test data are shown in Table 5.

[0120] Table 5:

[0121]

[0122] It can be understood that mAP combines Precision and Recall to comprehensively measure the detection effect of the model under different thresholds. The higher the mAP, the better the model performance. The calculation formulas for precision and recall are shown in formula (4) and formula (5):

[0123] Formula (4)

[0124] Formula (5)

[0125] In the formula, TP is the number of samples correctly judged as positive, TN is the number of samples correctly judged as negative, FP is the number of samples incorrectly judged as positive, and FN is the number of samples incorrectly judged as negative.

[0126] It can be understood that by calculating the average of the precision values ​​corresponding to each recall rate, the area under the accuracy and recall curve (AP) can be obtained. The average AP value of all categories is mAP. The calculation formulas of AP and mAP are shown in formula (6) and formula (7):

[0127] Formula (6)

[0128] Formula (7)

[0129] In the formula, is the i+1th different recall value, is the maximum precision at this recall rate, N is the number of categories, is the average precision of the i-th category.

[0130] It is understandable that F1 is a measurement indicator that comprehensively considers precision and recall. It is mostly applicable to classification problems of unbalanced data sets. It is the harmonic mean of precision and recall, and is used to weigh the accuracy and coverage of the model when predicting the positive class. The calculation formula is shown in formula (8):

[0131] Formula (8)

[0132] A training method for a neural network model provided by an embodiment of the present disclosure solves the problems of insufficient segmentation power of a small model, long reasoning time of a large model, and high overfitting rate. Small models usually have defects such as inaccuracy, unclear boundaries, jaggedness, and mis-segmentation for tongue segmentation, and large models usually have defects such as slow reasoning time, high overfitting rate, low generalization, and weak scalability. The tongue detection model and tongue segmentation model trained in this application can achieve high-precision tongue segmentation, and greatly reduce the model reasoning time. At the same time, it can adapt to the input of multiple scenes, and has strong segmentation ability and wide applicability. At the same time, it also solves the problem that a large amount of labeled data is required for the tongue segmentation task, and the prompt word prompt is input to the tongue segmentation model, so that the model pre-trained on the data set has the generalization ability of a small number of samples. In addition, using a standard color card to correct the image can effectively reduce the influence of ambient light on the color and brightness of the image, ensure that the image has consistent color and brightness characteristics under different acquisition conditions, and improve the image quality by removing noise, ensure the consistency of input data, reduce the error and noise interference caused by image quality differences, and improve the training accuracy of the model.

[0133] Based on the above embodiments, Fig.11 The process diagram of the tongue image processing method provided by the embodiment of the present disclosure specifically includes the following steps: Fig.11 Steps shown:

[0134] S110, obtaining a tongue image to be processed.

[0135] The tongue image includes a tongue object and a background object.

[0136] It can be understood that the tongue object to be processed is obtained, and the tongue object includes the tongue body object to be segmented and the background object, and the background object can be a part of a human face.

[0137] Optionally, obtaining the tongue image to be processed in the above S110 can be implemented by the following steps:

[0138] The original image is obtained, and the original image is corrected using a standard color card so that the corrected original image has a color and brightness that matches the input data corresponding to the tongue detection model and the tongue segmentation model; the corrected original image is subjected to noise removal processing and / or image brightness standardization processing to obtain a processed original image; the processed original image is converted into a color space corresponding to the tongue detection model and the tongue segmentation model by a color space conversion technology to obtain a tongue image to be processed.

[0139] It is understandable that the original image is obtained, and the original image refers to the image directly collected. The original image is color corrected by a standard color card so that the original image has a color and brightness that matches the input data of the tongue detection model and the tongue segmentation model, that is, the original image is preprocessed with the same operation as the above-mentioned detection training data to ensure the detection accuracy of the model. Subsequently, the original image is processed by removing noise, correcting color, and standardizing image brightness to significantly improve the quality of the image. The original image can also be converted to a color space corresponding to the model by color space conversion technology to obtain a tongue image to be processed. Preprocessing the original image helps to ensure the consistency of the input data, reduce errors and noise interference caused by differences in image quality, and ensure that the input data is consistent with the distribution of the data set used for training in actual applications. The specific implementation process of preprocessing is shown in the above embodiment, which will not be repeated here.

[0140] S111. Use the tongue image as the input of a pre-trained tongue detection model, detect the tongue object in the tongue image, and output the detection result.

[0141] The detection result includes detection information of the detection area corresponding to the tongue object in the tongue image, a detection image obtained by explicitly marking the detection area in the tongue image, and a cropped image of the detection area.

[0142] It can be understood that, based on the above S110, the tongue image is used as the input of the tongue detection model, and the tongue detection model is used to identify the shape, color, texture and other features of the tongue object in the tongue image, accurately locate and detect the tongue object in the tongue image, and output the detection result. Among them, the detection result includes a detection image, detection information and a cropped image. The detection image is a detection area in which the tongue object is marked in the tongue image. The detection image retains the original image information of the tongue image and explicitly marks the detection frame (detection area) in the tongue image. The detection information is a label information file (.txt file). The detection information includes the coordinates (x, y) of the target pixel points in the detection frame of the tongue object in the detection image and the width and height (w, h) of the detection frame. The cropped image is a cropped image of the detection frame area. The cropped image is cropped from the original tongue image and retains the image information inside the detection frame.

[0143] Optionally, in the above S111, the tongue image is used as the input of a pre-trained tongue detection model, the tongue object in the tongue image is detected, and the detection result is output, which can be specifically implemented by the following steps:

[0144] The tongue image is classified and detected by the tongue detection model, and when it is determined that the type of the tongue image is a tongue, the detection information of the detection area corresponding to the tongue object in the tongue image is output, wherein the detection information includes the width and height of the detection area and the coordinate information of the target pixel point in the detection area in the tongue image; based on the detection information, a bounding box of the detection area is explicitly marked in the tongue image to obtain a detection image, and the detection area is cropped out in the tongue image to obtain a cropped image.

[0145] It can be understood that the tongue image is classified and detected by the tongue detection model to determine the type of the tongue image. When it is determined that the type of the tongue image is a tongue, the detection information of the detection area corresponding to the tongue object in the tongue image is output, wherein the detection information includes the width and height of the detection area and the coordinate information of the target pixel point in the detection area in the tongue image, and the target pixel point can be the vertex or center point of the detection box. Subsequently, based on the detection information, the boundary box of the detection area is explicitly marked in the tongue image to obtain a detection image, and the detection area is cropped out in the tongue image to obtain a cropped image. The specific implementation process of the tongue detection model refers to the above embodiment and will not be repeated here.

[0146] S112: Use the cropped image as input to a pre-trained tongue segmentation model, segment the tongue object in the cropped image, and output a tongue segmentation map.

[0147] The tongue segmentation map includes original pixel points of the tongue object in the tongue image and background pixel points whose pixel values ​​are preset values.

[0148] It can be understood that, based on the above S111, the cropped image is used as the input of the tongue segmentation model, reasoning is performed through the tongue segmentation model, and a tongue segmentation map is output, which includes the original pixel points of the tongue object in the tongue image and the background pixel points with pixel values ​​of preset values, that is, the tongue object is represented by the original pixel points in the cropped image, and the background object is represented by the pixel points with preset values. While retaining the original information of the tongue object, the tongue object and the background object are distinguished. For example, the tongue segmentation map contains the pixel points of the tongue object in the original tongue image or cropped image and the pixel points with pixel values ​​of (0, 0, 0). The specific implementation process of the tongue segmentation model refers to the above embodiment and will not be repeated here.

[0149] S113. According to the detection information, the visualized tongue segmentation map and the detection image are spliced ​​together, and the spliced ​​detection image is displayed.

[0150] It is understandable that, based on the above S112, the tongue segmentation map is visualized to visually enhance the tongue object so as to distinguish it from the background object in the tongue image. Subsequently, the tongue segmentation image is drawn in the detection area in the detection image, and the drawn detection image is displayed. The tongue segmentation map can also be saved separately to facilitate other processing and judgment of the tongue segmentation map, such as using the tongue segmentation map to judge the tongue color, extract the tongue coating, and so on.

[0151] Optionally, in the above S113, according to the detection information, the visualized tongue segmentation image and the detection image are spliced, and the spliced ​​detection image is displayed, which can be specifically implemented by the following steps:

[0152] The original pixel points of the tongue object included in the tongue segmentation map are visualized to visually enhance the tongue object, wherein the visualization processing includes RGB transformation; based on the detection information, the visually enhanced tongue object is spliced ​​to the corresponding position in the detection area in the detection image, and the pixel points at the corresponding position are hidden to obtain a spliced ​​image; and at least one of the tongue image, the spliced ​​image, the detection image and the tongue segmentation map is displayed.

[0153] Understandably, the tongue segmentation map is transformed into RGB. The RGB transformation can enhance the visual effect of the segmented image, making the segmentation result more intuitive and explainable. After the RGB transformation, the segmentation result is easier to observe and analyze, which helps users understand more quickly. Specifically, the tongue segmentation map can be converted into a highlighted tongue segmentation map. Subsequently, the complete detection image, detection information, and highlighted tongue segmentation map are used as inputs. The image coordinates (x, y) in the detection information and the width and height (w, h) of the detection frame are used to draw the highlighted tongue segmentation map at the corresponding position in the complete detection image to obtain a complete highlighted tongue segmentation map. At the same time, the pixels covered by the tongue segmentation image can also be hidden.

[0154] For example, see Fig.12 , Fig.12 The tongue segmentation model reasoning flow chart provided for the embodiment of the present disclosure, after obtaining the image data (tongue image), it is necessary to perform a preprocessing operation on the image data to obtain preprocessed data to ensure that the data is consistent with the distribution of the data set used during training. Subsequently, the trained tongue detection model is used as a data processing model, and the preprocessed data is input into the tongue detection model, so that it outputs a complete detection image (detection image), tongue detection frame information (detection information) and a tongue area map (cropped image), completing the reasoning stage of the tongue detection model. The pre-trained tongue segmentation model is used as a data processing model, and the tongue area map output by the tongue detection model is used as input, and the tongue segmentation model is used for segmentation processing to obtain a tongue segmentation map. Finally, the complete tongue detection map, tongue detection frame information, and tongue segmentation map are output to the visualization module to complete the reasoning stage of the tongue segmentation model.

[0155] For example, see Fig.13 , Fig.13 The visualization process flow chart of the tongue segmentation result provided in the embodiment of the present disclosure is as follows: Fig.12 The processing of the model reasoning stage is shown in the figure, and the visualization stage is carried out. The visualization stage mainly realizes data integration and presents it in the form of pictures. The specific process is: the tongue segmentation map output by the model reasoning stage is saved separately for post-processing; the tongue segmentation map is transformed into RGB to obtain a highlighted tongue segmentation map, so that the display of the tongue object is more obvious. The highlighted tongue segmentation map includes the highlighted area and background area of ​​the tongue object, that is, the area where the tongue object is located is highlighted; the highlighted tongue segmentation map after RGB transformation is combined with the complete tongue detection map and the detection frame information, and the highlighted tongue segmentation map is re-spliced ​​into the complete tongue detection map, that is, the highlighted tongue object is used to replace the original tongue object in the detection frame to complete the display of the complete highlighted tongue segmentation map.

[0156] The disclosed embodiment provides a method for processing tongue images, which uses a tongue detection model to frame the tongue position and detection area, and crops the detection area, and uses the cropped image as the input of the tongue segmentation model to accurately segment the tongue object to obtain a tongue segmentation map. The interaction between the tongue detection model and the tongue segmentation model solves the problems of inaccurate tongue segmentation, unclear boundaries, jagged shapes, high mis-segmentation rate, and slow tongue segmentation reasoning time, high overfitting rate, low generalization, and weak scalability of the large model, thereby improving the accuracy, generalization ability, and scalability of tongue segmentation and greatly reducing the reasoning time of the model. Secondly, using the cropped image output by the tongue detection model as input reduces the reasoning time of the tongue segmentation model, and the resolution of the cropped image is also greatly reduced compared to the original tongue image. The image size resolution is reduced to about 3.67% of the original size on average, which greatly reduces the reasoning time of the tongue segmentation model. In addition, a complete segmentation map is inverted using the detection information, detection image and tongue segmentation map. That is, a small-sized tongue segmentation map can be drawn to the corresponding position in the detection image by combining the detection frame information in the detection information. This solves the problem that after segmentation using the cropped small-sized area map, only a small-sized tongue segmentation map can be obtained, and the segmentation result cannot be displayed in the original tongue image.

[0157] Based on the above embodiments, Fig.14The flowchart of the interactive tongue image processing method based on the tongue detection model and the tongue segmentation model provided in the embodiment of the present disclosure is as follows. The complete tongue image processing method mainly includes three stages: model training stage, model reasoning stage and visualization stage. Specifically, 1) the model training stage is mainly to prepare for the model reasoning stage. In this stage, the image data required by the algorithm will be collected, and the image data will be preprocessed and enhanced. At the same time, the tongue detection model and the tongue segmentation model data sets will be produced respectively. Subsequently, the produced data sets will be used to train the tongue detection model and the tongue segmentation model respectively to obtain the weight files of each model in preparation for the model reasoning stage. Among them, the model The training phase includes image data acquisition, image data preprocessing, detection training data annotation, image data enhancement, tongue detection model training, tongue image cropping, segmentation data annotation and tongue segmentation model training. Image data acquisition is used to collect high-quality tongue image data and obtain a rich and diverse data set, which can significantly improve the generalization ability and robustness of the model, so that it can still maintain high-precision detection and recognition performance under different environments and conditions; image data preprocessing is used to preprocess the collected image data to improve the image quality. Data preprocessing also helps to ensure the consistency of input data, reduce errors and noise interference caused by differences in image quality, and at the same time, high quality The preprocessed images can provide more accurate and reliable training data for the model, reduce data noise and outlier interference during model training, and help the model better learn image features, thereby improving the detection and segmentation performance of the model; detection training data annotation is used to annotate the preprocessed images. Data annotation provides the necessary supervision information for deep learning models. Through annotated data, the model can learn the characteristics and patterns of specific targets, which is crucial for training effective detection models. High-quality annotated data can significantly improve the accuracy and reliability of the model, making it perform better in practical applications; image data enhancement is used to enhance the annotated images. Data enhancement generates a large number of new training samples by performing different forms of transformations on the original images (such as rotation, flipping, scaling, cropping, color adjustment, etc.). This method can effectively expand the scale of the training data set, solve the problem of insufficient data, and improve the learning effect of the model. By performing different forms of transformations on the images, data enhancement can increase the diversity of the data, enabling the model to adapt to more diverse inputs, helping to improve the performance of the model on unseen test data, enhance its generalization ability, and prevent overfitting. Tongue detection model training is used to train the pre-trained tongue detection model using the tongue detection data set to obtain a detection model that can accurately identify and locate the tongue.Tongue image cropping is used to detect and locate the tongue in the collected image data through the tongue detection model, mark the specific position and area of ​​the tongue, obtain a complete detection image and detection frame information, and crop the detection frame to obtain a cropped image; segmentation data annotation is used to annotate the cropped image with segmentation model data. The segmentation image annotation is the pixel-level annotation provided by the model, which provides the necessary supervision information for the supervised learning model, so that the model can learn the tongue features such as the precise boundary features and shape features of the tongue, and can continuously adjust the parameters during the training process to optimize the model performance and finally achieve accurate image segmentation; tongue segmentation model training is used to take the tongue segmentation dataset as input, fine-tune the pre-trained tongue segmentation model, so that it can adapt to the tongue segmentation task, and obtain a tongue segmentation model that can accurately segment the tongue. ;

[0158] 2) Model inference stage: The required weight files of the tongue detection model and the tongue segmentation model have been obtained in the model training stage. In this stage, the tongue detection model and the tongue segmentation model will be used to complete the image inference task and prepare data for the subsequent visualization stage. The model inference stage includes tongue detection model inference and tongue segmentation model inference. The tongue detection model inference is to input the image that actually needs to be processed into the tongue detection model. The model will infer the input image and output the detection information, detection map and cropped image. The cropped image will be used in the subsequent tongue segmentation model inference; the tongue segmentation model inference is to use the cropped image as input, use the tongue segmentation model for inference, and output the segmentation mask map (tongue segmentation map). The segmentation mask map only contains the tongue object pixels in the original image and the pixels with pixel values ​​of (0, 0, 0). The segmentation mask map will be used in the subsequent visualization stage.

[0159] 3) Visualization stage: After processing in the model inference stage, the image data and text data required for visualization are obtained. The visualization stage mainly integrates the data and presents it in the form of pictures. Specifically, the visualization stage includes the preservation of the tongue segmentation map, the RGB transformation of the tongue segmentation map, and the drawing of a complete highlighted tongue segmentation map. The preservation of the tongue segmentation map is to save the segmentation mask map separately, which is convenient for subsequent processing and judgment of the tongue image, such as using the segmentation map for tongue color judgment, tongue coating extraction, etc.; the RGB transformation of the tongue segmentation map is to transform the segmentation mask map into RGB. The RGB transformation can enhance the visual effect of the segmented image and make the segmentation result more intuitive and explainable. After the RGB transformation, the segmentation result is easier to observe and analyze, which helps users to understand more quickly. The tongue segmentation map can be converted into a highlighted tongue segmentation map for output; the drawing of the complete highlighted tongue segmentation map is to take the complete detection image, detection information and highlighted tongue segmentation map as input, and use the image coordinates (x, y) in the detection information and the width and height (w, h) of the detection box to draw the highlighted tongue segmentation map at the corresponding position in the complete detection image to obtain a complete highlighted tongue segmentation map.

[0160] Fig.15 The structure diagram of the tongue image processing device provided in the embodiment of the present disclosure is as follows. The tongue image processing device provided in the embodiment of the present disclosure can execute the processing flow provided in the tongue image processing method embodiment, such as Fig.15 As shown, the tongue image processing device 1500 includes an acquisition unit 1510, a detection unit 1511, a segmentation unit 1512 and a display unit 1513, wherein:

[0161] An acquisition unit 1510 is used to acquire a tongue image to be processed, wherein the tongue image includes a tongue body object;

[0162] The detection unit 1511 is used to use the tongue image as an input of a pre-trained tongue detection model, detect the tongue object in the tongue image, and output a detection result, wherein the detection result includes detection information of a detection area corresponding to the tongue object in the tongue image, a detection image obtained by explicitly marking the detection area in the tongue image, and a cropped image of the detection area;

[0163] The segmentation unit 1512 is used to use the cropped image as an input of a pre-trained tongue segmentation model, segment the tongue object in the cropped image, and output a tongue segmentation map;

[0164] The display unit 1513 is used to splice the tongue segmentation map after visualization and the detection image according to the detection information, and display the spliced ​​detection image.

[0165] Optionally, the acquiring unit 1510 is configured to:

[0166] Acquire an original image, and correct the original image using a standard color card so that the corrected original image has a color and brightness that matches the corresponding input data of the tongue detection model and the tongue segmentation model;

[0167] Performing noise removal processing and / or image brightness standardization processing on the corrected original image to obtain a processed original image;

[0168] The processed original image is converted into a color space corresponding to the tongue detection model and the tongue segmentation model through the color space conversion technology to obtain the tongue image to be processed.

[0169] Optionally, the detection unit 1511 is used for:

[0170] Classify and detect the tongue image through the tongue detection model, and when it is determined that the type of the tongue image is a tongue, output the detection information of the detection area corresponding to the tongue object in the tongue image, wherein the detection information includes the width and height of the detection area and the coordinate information of the target pixel point in the detection area in the tongue image;

[0171] Based on the detection information, a bounding box of the detection area is explicitly marked in the tongue image to obtain a detection image, and the detection area is cropped out in the tongue image to obtain a cropped image.

[0172] Optionally, the display unit 1513 is used for:

[0173] Performing visualization processing on the tongue segmentation image including original pixel points of the tongue object to visually enhance the tongue object, wherein the visualization processing includes RGB transformation;

[0174] According to the detection information, the visually enhanced tongue object is spliced ​​to the corresponding position in the detection area of ​​the detection image, and the pixel points at the corresponding position are hidden to obtain a spliced ​​image;

[0175] At least one of the tongue image, the spliced ​​image, the detection image and the tongue segmentation image is displayed.

[0176] Optionally, the tongue image processing device 1500 is further used for:

[0177] Acquire detection training data, the detection training data includes multiple types of training data, each of which includes multiple tongue sample images, and the multiple types of training data include first-category training data, second-category training data, third-category training data, and fourth-category training data, the first-category training data includes a series of tongue sample images collected following a tongue object, the second-category training data includes consistent tongue sample images collected under the same shooting conditions and ambient lighting, the third-category training data includes diverse tongue sample images collected under different ambient lighting and / or shooting angles, and the fourth-category training data includes tongue sample images collected in a target application scenario;

[0178] The pre-built tongue detection model is trained according to the detection training data to obtain a trained tongue detection model;

[0179] According to the detection sample information of each tongue sample image output by the tongue detection model, the detection area in each tongue sample image is cropped out to obtain segmentation training data;

[0180] The pre-built tongue segmentation model is trained according to the segmentation training data to obtain a trained tongue segmentation model.

[0181] Optionally, the tongue image processing device 1500 is further used for:

[0182] Preprocessing a plurality of tongue sample images included in the detection training data to determine the color and brightness and color space of the input data corresponding to the tongue detection model and the tongue segmentation model;

[0183] Annotate the detection area corresponding to the tongue object in each tongue sample image to obtain detection annotation information of each tongue sample image;

[0184] Performing data enhancement processing on tongue sample images and / or detection annotation information to expand the detection training data of the tongue detection model;

[0185] Predicting the detection area of ​​the tongue object in the tongue sample image by using the tongue detection model to obtain detection sample information;

[0186] The first loss is calculated according to the coordinate information of each pixel point in the detection area included in the detection annotation information and the detection sample information in the tongue sample image, and the parameters of the tongue detection model are adjusted according to the first loss.

[0187] Optionally, the tongue image processing device 1500 is further used for:

[0188] Performing pixel annotation on the tongue object in each cropped sample image to obtain segmentation annotation information of each cropped sample image, wherein the multiple pixel points annotated in the segmentation annotation information represent the boundary and shape features of the tongue object;

[0189] Setting prompt information that the segmentation object is a tongue object, and segmenting the tongue object in the cropped sample image based on the prompt information by using the tongue segmentation model to obtain a segmented sample image;

[0190] The block loss and the cross entropy loss are combined to calculate the second loss according to the pixels in the segmented sample image and the segmented annotation information, and the parameters of the tongue segmentation model are adjusted according to the second loss.

[0191] Fig.15 The tongue image processing device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment, and its implementation principle and technical effect are similar and will not be described in detail here.

[0192] Fig.16 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig.16 , which shows a schematic diagram of the structure of an electronic device 1600 suitable for implementing the embodiment of the present disclosure. The electronic device 1600 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), wearable electronic devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Fig.16 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0193] like Fig.16 As shown, the electronic device 1600 may include a processing device 1601 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1602 or a program loaded from a storage device 1608 to a random access memory (RAM) 1603 to implement the tongue image processing method of the embodiment described in the present disclosure. In the RAM 1603, various programs and data required for the operation of the electronic device 1600 are also stored. The processing device 1601, the ROM 1602, and the RAM 1603 are connected to each other via a bus 1604. An input / output (I / O) interface 1605 is also connected to the bus 1604.

[0194] Typically, the following devices may be connected to the I / O interface 1605: an input device 1606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1609. The communication device 1609 may allow the electronic device 1600 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.16 The electronic device 1600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0195] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby implementing the tongue image processing method as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1609, or installed from the storage device 1608, or installed from the ROM 1602. When the computer program is executed by the processing device 1601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0196] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0197] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0198] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0199] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0200] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0201] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0202] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0203] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0204] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0205] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or gateway that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or gateway. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or gateway that includes the elements.

[0206] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing a tongue image, characterized in that: include: Acquire a tongue image to be processed, wherein the tongue image includes a tongue body object; The tongue image is used as an input of a pre-trained tongue detection model, the tongue object in the tongue image is detected, and a detection result is output, wherein the detection result includes detection information of a detection area corresponding to the tongue object in the tongue image, a detection image obtained by explicitly marking the detection area in the tongue image, and a cropped image of the detection area, wherein the detection information includes the width and height of the detection area and the coordinate information of the target pixel point in the detection area in the tongue image, and the cropped image is obtained by cropping the detection area in the tongue image; Using the cropped image as input of a pre-trained tongue segmentation model, segmenting the tongue object in the cropped image, and outputting a tongue segmentation map; According to the detection information, the tongue segmentation image after visualization processing and the detection image are spliced, and the spliced ​​detection image is displayed, wherein the visualization processing includes RGB conversion; The tongue detection model and the tongue segmentation model are trained by the following steps: Acquire detection training data, wherein the detection training data includes multiple types of training data, each of which includes multiple tongue sample images, and the multiple types of training data include first-category training data, second-category training data, third-category training data, and fourth-category training data, wherein the first-category training data includes a series of tongue sample images collected following a tongue object, the second-category training data includes consistent tongue sample images collected under the same shooting conditions and ambient lighting, the third-category training data includes diverse tongue sample images collected under different ambient lighting and / or shooting angles, and the fourth-category training data includes tongue sample images collected in a target application scenario; Training a pre-built tongue detection model according to the detection training data to obtain a trained tongue detection model; According to the detection sample information of each tongue sample image output by the tongue detection model, the detection area in each tongue sample image is cropped out to obtain segmentation training data; The pre-constructed tongue segmentation model is trained according to the segmentation training data to obtain a trained tongue segmentation model, wherein the tongue segmentation model is a SAM model.

2. The method according to claim 1, characterized in that The step of obtaining the tongue image to be processed comprises: Acquire an original image, and correct the original image using a standard color card so that the corrected original image has a color and brightness that matches the corresponding input data of the tongue detection model and the tongue segmentation model; Performing noise removal processing and / or image brightness standardization processing on the corrected original image to obtain a processed original image; The processed original image is converted into a color space corresponding to the tongue detection model and the tongue segmentation model by using a color space conversion technology to obtain a tongue image to be processed.

3. The method according to claim 1, characterized in that The method of using the tongue image as an input of a pre-trained tongue detection model, detecting the tongue object in the tongue image, and outputting the detection result includes: Classifying and detecting the tongue image by using the tongue detection model, and outputting detection information of a detection area corresponding to the tongue object in the tongue image when determining that the type of the tongue image is a tongue; Based on the detection information, a boundary box of the detection area is explicitly marked in the tongue image to obtain a detection image, and the detection area is cropped out in the tongue image to obtain a cropped image.

4. The method according to claim 1, characterized in that: The tongue segmentation map includes original pixel points of the tongue object in the tongue image and background pixel points whose pixel values ​​are preset values. The visually processed tongue segmentation map and the detection image are spliced ​​according to the detection information, and the spliced ​​detection image is displayed, including: Performing visualization processing on original pixel points of the tongue object included in the tongue segmentation map to visually enhance the tongue object; splicing the visually enhanced tongue object to a corresponding position in the detection area of ​​the detection image according to the detection information, and hiding the pixel points at the corresponding position to obtain a spliced ​​image; At least one of the tongue image, the spliced ​​image, the detection image and the tongue segmentation image is displayed.

5. The method according to claim 1, characterized in that The step of training the pre-built tongue detection model according to the detection training data to obtain a trained tongue detection model includes: Preprocessing a plurality of tongue sample images included in the detection training data to determine the color and brightness and color space of the input data corresponding to the tongue detection model and the tongue segmentation model; Annotate the detection area corresponding to the tongue object in each tongue sample image to obtain detection annotation information of each tongue sample image; Performing data enhancement processing on the tongue sample image and / or the detection annotation information to expand the detection training data of the tongue detection model; Predicting the detection area of ​​the tongue object in the tongue sample image by using the tongue detection model to obtain detection sample information; A first loss is calculated according to the coordinate information of each pixel point in the detection area included in the detection annotation information and the detection sample information in the tongue sample image, and the parameters of the tongue detection model are adjusted according to the first loss.

6. The method according to claim 1, characterized in that The segmentation training data includes a plurality of cropped sample images, and the pre-built tongue segmentation model is trained according to the segmentation training data to obtain a trained tongue segmentation model, including: Performing pixel annotation on the tongue object in each cropped sample image to obtain segmentation annotation information of each cropped sample image, wherein the multiple pixel points annotated in the segmentation annotation information represent the boundary and shape features of the tongue object; Setting prompt information that the segmentation object is a tongue object, and segmenting the tongue object in the cropped sample image based on the prompt information by the tongue segmentation model to obtain a segmented sample image; A second loss is calculated based on the pixel points in the segmented sample image and the segmented annotation information by combining block loss and cross entropy loss, and the parameters of the tongue segmentation model are adjusted based on the second loss.

7. A tongue image processing device, characterized in that: include: An acquisition unit, used for acquiring a tongue image to be processed, wherein the tongue image includes a tongue body object; a detection unit, configured to use the tongue image as an input of a pre-trained tongue detection model, detect the tongue object in the tongue image, and output a detection result, wherein the detection result includes detection information of a detection area corresponding to the tongue object in the tongue image, a detection image obtained by explicitly marking the detection area in the tongue image, and a cropped image of the detection area, wherein the detection information includes a width and a height of the detection area and coordinate information of a target pixel point in the detection area in the tongue image, and the cropped image is obtained by cropping the detection area in the tongue image; A segmentation unit, used for taking the cropped image as an input of a pre-trained tongue segmentation model, segmenting the tongue object in the cropped image, and outputting a tongue segmentation map; A display unit, used for splicing the tongue segmentation map after visualization processing and the detection image according to the detection information, and displaying the spliced ​​detection image, wherein the visualization processing includes RGB conversion; The tongue detection model and the tongue segmentation model are trained by the following steps: Acquire detection training data, wherein the detection training data includes multiple types of training data, each of which includes multiple tongue sample images, and the multiple types of training data include first type of training data, second type of training data, third type of training data and fourth type of training data, wherein the first type of training data includes a series of tongue sample images collected following the tongue object, the second type of training data includes consistent tongue sample images collected under the same shooting conditions and ambient lighting, the third type of training data includes diverse tongue sample images collected under different ambient lighting and / or shooting angles, and the fourth type of training data includes tongue sample images collected in a target application scenario; train a pre-constructed tongue detection model according to the detection training data to obtain a trained tongue detection model; crop the detection area in each tongue sample image according to the detection sample information of each tongue sample image output by the tongue detection model to obtain segmentation training data; train a pre-constructed tongue segmentation model according to the segmentation training data to obtain a trained tongue segmentation model, wherein the tongue segmentation model is a SAM model.

8. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the tongue image processing method as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the tongue image processing method according to any one of claims 1 to 6 are implemented.