Tongue picture precise segmentation system based on SAM model

Through the precise segmentation system of tongue image based on SAM model, using image acquisition, processing and pre-training models, the problem that traditional tongue image diagnosis relies on doctors' experience is solved, and the precise segmentation and feature extraction of tongue image images is realized, which improves the accuracy of diagnosis. and reliability enhance the robustness and generalization capabilities of the model.

CN119067984APending Publication Date: 2024-12-03SHENZHEN HUIYIBIDA MEDICAL TECH CO LTD

Patent Information

Application Number
CN202411234575.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-12-03

Smart Images

  • Figure CN119067984A_ABST
    Figure CN119067984A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image processing, in particular to a tongue picture precise segmentation system based on an SAM model. According to the technical scheme, the segmentation system comprises an image acquisition module, an image processing module and a pre-training model, the image acquisition module is constructed to record and encode tongue images, the image processing module is constructed to process the recorded images, and the pre-training model is constructed to output image features. Image data recorded by the image acquisition module is transmitted to the inside of the image processing module for enhancement, the image acquisition module, the image processing module and the pre-training model are arranged, so that the effect of improving the data processing quality is achieved, multiple regions such as a tongue body and a tongue coating in a tongue image can be accurately segmented, and the accuracy of the tongue image is improved. And moreover, useful features are extracted from the tongue picture image and are subjected to quantitative analysis, so that the generalization ability and the practicability of the model can be ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a tongue image precise segmentation system based on a SAM model. Background Art

[0002] Tongue diagnosis plays an important role in TCM theory. It is to judge the internal health of the human body by observing the color, shape, thickness of tongue coating and other characteristics of the tongue. As an important part of TCM visual diagnosis, the accuracy and objectivity of tongue image are directly related to the reliability of the diagnosis results. SAM is a large-scale pre-trained interactive segmentation model, known for its powerful zero-shot generalization ability. Applying SAM to tongue segmentation can achieve zero-shot segmentation of various types of tongue images. The original SAM model uses ViT-H as the image encoder, with more than 632M parameters. This is a large model, which leads to a significant increase in model training and prediction time.

[0003] Patent announcement number CN109087313A discloses an intelligent tongue segmentation method based on deep learning, involving image processing. Formulate tongue image acquisition standards to collect tongue image data under standard environment; the acquisition standards include equipment, light, location and patient requirements; manually segment tongue image data, and divide the data set into training set and test set to verify the effectiveness of the algorithm; obtain low-level features of tongue image and perform pixel positioning through the encoder of tongue segmentation network TS-Net; fuse high-level and low-level features and perform pixel classification through the decoder of tongue segmentation network TS-Net to obtain the complete tongue area.

[0004] Although the existing technology has many benefits during use, it still has the following problems: the traditional tongue diagnosis method mainly relies on the doctor's experience and subjective judgment, which is easily affected by factors such as ambient light and doctor's fatigue, resulting in deviations and errors in the diagnosis results. In order to improve the accuracy and reliability of tongue diagnosis, the use of artificial intelligence technology for accurate segmentation and analysis of tongue images has become a research hotspot. Summary of the invention

[0005] The purpose of the present invention is to provide a tongue image accurate segmentation system based on the SAM model to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions: a tongue image accurate segmentation system based on the SAM model, the segmentation system includes an image acquisition module, an image processing module and a pre-training model; The image acquisition module is constructed to record and encode tongue images, the image processing module is constructed to process the recorded images, and the pre-trained model is constructed to output image features. The image data recorded by the image acquisition module is transmitted to the image processing module for enhancement, and the enhanced image data is transmitted to the pre-trained model for training.

[0007] Preferably, the image acquisition module also includes a tongue image recording module, which uses a camera to adjust the recording distance and records the tongue image, and stores and records the tongue image after the tongue image is acquired.

[0008] Preferably, the image acquisition module also includes an image encoding module, which uses an encoder to compile the collected tongue image data and transmits the compiled image data to the pre-trained model.

[0009] Preferably, the image processing module automatically generates a suggestive rectangular box based on the labeled mask image and performs accurate processing. When predicting, the Faster RCNN target detection model is used to estimate the target position rectangular box, and a bounding box training strategy is adopted when training the segmentation model; First, determine a perturbation factor factor. When using the mask map to automatically generate a bounding box, the key data obtained include: the upper left corner point (x_min, y_min) and lower right corner point (x_max, y_max) of the bounding box, as well as the width W and height H of the image.

[0010] Preferably, the pre-trained model also includes a lightweight training model. The lightweight training model refers to the design of the MobileSAM model. By distilling the original ViT of the SAM model, a lightweight ViT-Tiny can be obtained as image encoder_1.

[0011] Preferably, the pre-trained model also includes a high-quality training model, which draws on the EfficientSAM model and uses the MAE pre-training method and the SAM mask image pre-training to obtain a high-quality pre-trained ViT-Tiny as image encoder_2; The pre-trained model is transferred to the lightweight training model and the high-quality training model respectively. The final output result is the sum of the inputs of the two encoders. The two encoders can learn image features from different angles, which enhances the robustness of the model. Even if one encoder performs poorly in some cases, the other encoder may still provide useful information.

[0012] Compared with the prior art, the beneficial effects of the present invention are: the present invention achieves the effect of improving data processing quality by setting an image acquisition module, an image processing module and a pre-trained model, and can accurately segment the tongue body and tongue coating in multiple areas in the tongue image for subsequent feature analysis, and extract useful features from the tongue image and perform quantitative analysis, which can ensure the generalization ability and practicality of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] See also Figure 1 , the present invention provides three embodiments: Embodiment 1: A tongue image accurate segmentation system based on the SAM model, the segmentation system includes an image acquisition module, an image processing module and a pre-training model; The image acquisition module is constructed to record and encode tongue images, the image processing module is constructed to process the recorded images, and the pre-trained model is constructed to output image features. The image data recorded by the image acquisition module is transmitted to the image processing module for enhancement, and the enhanced image data is transmitted to the pre-trained model for training.

[0016] The image acquisition module also includes a tongue image recording module, which uses a camera to adjust the recording distance and records the tongue image, and stores and records the tongue image after the tongue image is collected.

[0017] The image acquisition module also includes an image encoding module, which uses an encoder to compile the collected tongue image data and transmits the compiled image data to the pre-training model.

[0018] Embodiment 2: A tongue image accurate segmentation system based on the SAM model, the segmentation system includes an image acquisition module, an image processing module and a pre-training model; The image acquisition module is constructed to record and encode tongue images, the image processing module is constructed to process the recorded images, and the pre-trained model is constructed to output image features. The image data recorded by the image acquisition module is transmitted to the image processing module for enhancement, and the enhanced image data is transmitted to the pre-trained model for training.

[0019] The image acquisition module also includes a tongue image recording module, which uses a camera to adjust the recording distance and records the tongue image, and stores and records the tongue image after the tongue image is collected.

[0020] The image acquisition module also includes an image encoding module, which uses an encoder to compile the collected tongue image data and transmits the compiled image data to the pre-training model.

[0021] The image processing module automatically generates a suggestive rectangular box based on the labeled mask image and performs precise processing. When predicting, the Faster RCNN target detection model is used to estimate the target's position rectangular box. The target detection model may not be able to accurately locate the target rectangular box on an unknown data set. Therefore, when training the segmentation model, we adopted a specific bounding box training strategy; First, determine a perturbation factor factor. When using the mask map to automatically generate a bounding box, the key data obtained include: the upper left corner point (x_min, y_min) and lower right corner point (x_max, y_max) of the bounding box, as well as the width W and height H of the image.

[0022] The formula for perturbing the bounding box is as follows: .

[0023] This approach allows us to introduce some flexibility into the training process, thereby increasing the model's robustness to small changes in the boundary position. This step is critical because it helps the model adapt to errors that may occur due to insufficient object detection accuracy. We then use the perturbed bounding boxes as input to the model for prompt engineering, which improves the accuracy of tongue diagnosis and reduces human error through standardized image acquisition and precise tongue segmentation.

[0024] Embodiment 3: A tongue image accurate segmentation system based on the SAM model, the segmentation system includes an image acquisition module, an image processing module and a pre-trained model; The image acquisition module is constructed to record and encode tongue images, the image processing module is constructed to process the recorded images, and the pre-trained model is constructed to output image features. The image data recorded by the image acquisition module is transmitted to the image processing module for enhancement, and the enhanced image data is transmitted to the pre-trained model for training.

[0025] The image acquisition module also includes a tongue image recording module, which uses a camera to adjust the recording distance and records the tongue image, and stores and records the tongue image after the tongue image is collected.

[0026] The image acquisition module also includes an image encoding module, which uses an encoder to compile the collected tongue image data and transmits the compiled image data to the pre-training model.

[0027] The image processing module automatically generates a suggestive rectangular box based on the labeled mask image and performs precise processing. When predicting, the Faster RCNN target detection model is used to estimate the target position rectangular box, and a bounding box training strategy is adopted when training the segmentation model; First, determine a perturbation factor factor. When using the mask map to automatically generate a bounding box, the key data obtained include: the upper left corner point (x_min, y_min) and lower right corner point (x_max, y_max) of the bounding box, as well as the width W and height H of the image.

[0028] The pre-trained model also includes a lightweight training model. The lightweight training model refers to the design of the MobileSAM model. By distilling the original ViT of the SAM model, a lightweight ViT-Tiny can be obtained as image encoder_1. The training principle of Image Encoder_1: Distillation uses decoupled distillation (DecoupledDistillation), which directly distills knowledge from the encoder part of the original Vision Transformer to a smaller encoder without relying on the original Mask Decoder part. This method simplifies the distillation process, avoids the coupled optimization problem existing in traditional distillation, and can use a simpler mean square error (MSE) loss function.

[0029] in Figure 1 A is the model structure of Tongue-SAM, B is Image Encoder_1, and C is Image Encoder_2.

[0030] Prompt Guided Mask Decoder is a combination of Prompt, Prompt Encoder and Mask Decoder in part A.

[0031] The pre-trained model also includes a high-quality training model. The high-quality training model draws on the EfficientSAM model. By using the MAE pre-training method and the SAM mask image pre-training, a high-quality pre-trained ViT-Tiny can be obtained as image encoder_2. The training principle of Image Encoder_2 is: CrossAttention Decoder uses masked image tokens as queries in pre-training, combines the unmasked and masked features output by the encoder as keys and values, and reconstructs the masked tokens, helping ViT-Tiny learn feature representation capabilities similar to ViTBase Image Encoder, thereby improving its performance in visual representation and segmentation tasks; By introducing the idea of ​​divide and conquer algorithm, it is decided to lock the mask decoder, optimize the image encoder, and transfer the pre-trained model to the lightweight training model and the high-quality training model respectively. The final output result is the sum of the inputs of the two encoders. The two encoders can learn image features from different angles. This design of using two image encoders has its unique advantages, which enhances the robustness and generalization ability of the model. Even if one encoder performs poorly in some cases, the other encoder may still provide useful information. Such redundant design can improve the generalization ability of the model for different data distributions. In addition, since both encoders use lightweight ViT, the number of parameters has been greatly reduced. The automated tongue image analysis system eliminates the influence of doctors' subjective judgment and ensures the objectivity and consistency of the diagnosis results. The system can provide doctors with scientific tongue image feature data, assist doctors in making more comprehensive and accurate diagnoses, and improve the diagnosis and treatment effects. The automated analysis process saves doctors' time and energy and improves work efficiency, especially when facing a large number of patients. The system can accumulate a large amount of tongue image data, support medical research and big data analysis, and contribute to the development and progress of TCM tongue image diagnosis theory.

[0032] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A tongue image accurate segmentation system based on SAM model, Features: The segmentation system includes an image acquisition module, an image processing module and a pre-trained model; The image acquisition module is constructed to record and encode tongue images, the image processing module is constructed to process the recorded images, and the pre-trained model is constructed to output image features. The image data recorded by the image acquisition module is transmitted to the image processing module for enhancement, and the enhanced image data is transmitted to the pre-trained model for training.

2. According to claim 1, a tongue image accurate segmentation system based on the SAM model, Features: The image acquisition module also includes a tongue image recording module, which uses a camera to adjust the recording distance and records the tongue image, and stores and records the tongue image after the tongue image is collected.

3. According to claim 2, a tongue image accurate segmentation system based on the SAM model, Features: The image acquisition module also includes an image encoding module, which uses an encoder to compile the collected tongue image data and transmits the compiled image data to the pre-training model.

4. According to claim 1, a tongue image accurate segmentation system based on the SAM model, Features: The image processing module automatically generates a suggestive rectangular box based on the labeled mask image and performs precise processing. When predicting, the Faster RCNN target detection model is used to estimate the target position rectangular box, and a bounding box training strategy is adopted when training the segmentation model; First, determine a perturbation factor factor. When using the mask map to automatically generate a bounding box, the key data obtained include: the upper left corner point (x_min, y_min) and lower right corner point (x_max, y_max) of the bounding box, as well as the width W and height H of the image.

5. According to claim 1, a tongue image accurate segmentation system based on the SAM model, Features: The pre-trained model also includes a lightweight training model. The lightweight training model refers to the design of the MobileSAM model. By distilling the original ViT of the SAM model, a lightweight ViT-Tiny can be obtained as image encoder_1.

6. A tongue image accurate segmentation system based on the SAM model according to claim 5, Features: The pre-trained model also includes a high-quality training model. The high-quality training model draws on the EfficientSAM model and uses the MAE pre-training method and the SAM mask image pre-training to obtain a high-quality pre-trained ViT-Tiny as image encoder_2; The pre-trained model is transferred to the lightweight training model and the high-quality training model respectively. The final output result is the sum of the inputs of the two encoders. The two encoders can learn image features from different angles, which enhances the robustness of the model. Even if one encoder performs poorly in some cases, the other encoder may still provide useful information.

Citation Information

Patent Citations

  • An intelligent tongue segmentation method based on depth learning

    CN109087313A

Cited By

  • Image acquisition device for interrogation and image acquisition method for interrogation

    CN119564161A