A method for generating tongue image data enhancement based on DINO perception guidance

Abstract

This invention discloses a method for enhancing and generating tongue image data based on DINO perception guidance. The method includes the following steps: (1) acquiring real tongue images labeled with different tags, and using a frozen parameter pre-trained variational autoencoder to map them to a low-dimensional target tongue image latent space; (2) constructing a time-dependent velocity field network with a UNet architecture backbone and integrating adaptive group normalized modulation, and training the network with the mean square error loss of flow matching; (3) introducing a label-free distillation model DINO perception guidance. When the model training rounds and the flow path time step are greater than a preset threshold, the flow matching loss and the DINO perception loss are weighted to obtain the total loss, and the velocity field prediction network is updated by backpropagation; (4) in the inference sampling stage, combining a rescaling classifier-free guidance strategy, tongue images with different tags are synthesized. This invention provides high-quality generated data to address the problem of scarce tongue diagnosis samples and enhances the accuracy of intelligent tongue diagnosis.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence, particularly to the field of TCM intelligent diagnostic technology based on tongue image, specifically to a method for enhancing and generating tongue image data based on DINO perception guidance through flow matching. Background Technology

[0002] Tongue diagnosis is an important diagnostic method in Traditional Chinese Medicine (TCM), reflecting the physiological and pathological conditions of organs by observing the color, shape, and coating of the tongue. With the development of computer vision technology, computerized tongue diagnosis has effectively reduced the subjective bias of traditional diagnosis and improved the repeatability and standardization of medical diagnoses. Deep learning methods, through end-to-end learning, can adaptively extract highly nonlinear microscopic pathological textures and color changes, becoming the mainstream method for achieving high-precision tongue diagnosis.

[0003] However, the application of tongue diagnosis faces serious challenges due to the scarcity of clinical data and class imbalance. Due to limitations in clinical resources and ethical review requirements, a large-scale and diverse database of tongue images is lacking in the field. To address this data deficiency, existing studies typically employ traditional image enhancement methods (such as random cropping, random flipping, and color jitter). However, these techniques can only perform simple geometric transformations in pixel space, resulting in enhanced samples that are highly similar to the original images semantically, lacking generative diversity.

[0004] In recent years, generative models, especially Latent Diffusion Models (LDMs), have made progress in medical image generation. However, LDMs have significant limitations in tongue image generation: their highly compressed latent space tends to smooth out subtle pathological features (such as micro-ecchymosis and fissures), resulting in low class discrimination of generated tongue image samples; and the unconstrained generation process often introduces structural illusions. Furthermore, traditional classifier-free guidance (CFG) at higher guidance scales can lead to color oversaturation and variance distortion in the generated tongue images. Therefore, there is an urgent need for a novel, high-fidelity tongue image data generation method that can maintain the overall anatomical structure while accurately preserving microscopic pathological details. Summary of the Invention

[0005] To address the problems described in the background section, this invention proposes a method for enhancing and generating tongue image data based on DINO-aware guidance through flow matching.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] Design a method for enhancing and generating tongue image data based on DINO-aware guidance through stream matching, including the following steps:

[0008] S1. Obtain real tongue image data labeled with different categories of tags (such as different stages of diabetes, different degrees of fatty liver), and compress them into low-dimensional latent vectors after preprocessing using a pre-trained variational autoencoder (VAE).

[0009] S2, Construct a UNet flow matching velocity field prediction network based on Adaptive Group Normalization (AdaGN) modulation, and train the network using the flow matching mean square error loss as the basic loss function;

[0010] S3. In the model training phase, a perceptual guidance mechanism based on the unlabeled distillation model DINO is introduced. When the model training rounds and the flow path time step parameters are greater than the preset threshold, the semantic features of the tongue image generated by the UNet prediction latent vector and the real tongue image are extracted using the frozen DINO model and the mean square error is calculated. The flow matching loss and the DINO perceptual loss are weighted and summed to obtain the total loss. The velocity field prediction network UNet is updated by backpropagation.

[0011] S4 introduces a rescaled classifier-free CFG strategy in the model inference sampling stage to synthesize tongue images, alleviate color oversaturation and preserve realistic texture, thereby effectively enhancing and expanding TCM tongue image data.

[0012] Furthermore, in step S1, tongue image samples of different categories are prepared, all samples are adjusted to a uniform pixel size, and they are mapped to the tongue image latent vector (denoted as x1) by a pre-trained VAE encoder.

[0013] Furthermore, in step S2, the goal of the flow matching network is to learn a time-dependent velocity field v. θ (x t (t, y). Define Gaussian white noise x0∼ (0, I), the hidden state vector x under any time step t ∈ [0, 1]. t Following a linear probability path: x t =(1−t)x0+tx1. The loss function of the flow matching network is the mean square error (MSE) between the predicted velocity field and the actual target tongue image latent vector velocity field path (denoted as v=x1−x0):

[0014]

[0015] Predictive velocity field network The UNet architecture based on Adaptive Group Normalization (AdaGN) modulation is adopted. The time step (t) and tongue image category label (y) are jointly embedded and injected into the residual block through the AdaGN module. Affine transformations are used to map them into normalized feature maps, so that the network can better align the tongue image structural features under specific category labels.

[0016] Furthermore, in step S3: since the initial tongue image generated is mostly random noise, DINO perception guidance is only triggered after a set training period (e.g., training epochs > 50) and when the generated trajectory is close to the target tongue image data distribution (t > 0.5). Specifically, when triggered, the model determines the target tongue image based on the current intermediate latent vector (x). t ) and UNet predicts latent vectors (v θ (x t The final latent vector is derived from the extrapolation of ,t,y). =x t +(1−t)v θ (x t The final tongue image latent vector will be decoded to pixel space using a frozen VAE decoder to obtain the predicted tongue image; the predicted tongue image (t,y) ) and original real tongue image ( The semantic features are extracted from the frozen pre-trained DINO network (ϕ(⋅)), and the mean squared error between the two features is calculated as the perceptual loss to guide the perceptual loss L. DINO The calculation formula is as follows:

[0017]

[0018] The weighted sum of the mean squared error loss from the flow matching is used as the overall optimization objective for network training. The overall training objective is: L Total =L FM +λ DINO L DINO , where λ DINO Hyperparameters for controlling the intensity of perception guidance.

[0019] Furthermore, in step S4: During the inference phase, the tongue image latent vector is generated using a rescaled CFG. Starting from Gaussian white noise at the initial step size (t=0), the trained tongue image velocity field Unet prediction network is used to calculate the conditional velocity field (v) at each time step (t) at the current time step (t). cond ) and unconditional velocity field (v uncond The predicted values ​​are used to construct a standard CFG predicted velocity field v using the difference between the two values. cfg = v uncond + w⋅(vcond - v uncond ), where w is the guiding weight. To prevent overexposure of the tongue image, the standard deviation σ(v) of both the conditional and unconditional velocity fields is calculated. cond ) and σ(v cfg ), thus obtaining the rescaled velocity field v rescaled :

[0020]

[0021] Where ε is a minimal constant to prevent the denominator from being zero. An interpolation factor is introduced ( The velocity field at the current time step is obtained by mixing the standard CFG and the rescaled CFG velocity fields: v final= ⋅v rescaled +(1− )⋅v cfg Finally, within one integration time step Δt, the latent vector state is updated using the Euler method. =x t +v final ⋅Δt. The tongue image latent vector predicted by the UNet network at t=1 is input into the frozen pre-trained VAE decoder to generate a pixel-space tongue image.

[0022] This invention innovatively applies a latent space flow matching framework to the generation of TCM tongue images. Combined with the powerful unlabeled distillation model DINO semantic feature-guided loss, it effectively avoids the smoothing out of microscopic pathological details of the tongue image or the generation of structural illusions during latent space compression. Simultaneously, the rescaled CFG inference strategy effectively eliminates color distortion and overexposure of the tongue image under high guided scales. This method can achieve high-quality tongue image data augmentation without relying on massive amounts of clinically labeled samples, improving the performance of downstream tongue image-based intelligent diagnostic systems. It provides a novel paradigm for generating high-quality tongue image data to address the problems of data scarcity and class imbalance in intelligent diagnosis and treatment of TCM tongue images. Attached Figure Description

[0023] The accompanying drawings are part of this invention and are primarily intended to facilitate understanding and explanation of the invention. They should not be construed as undue limitations on the invention.

[0024] Figure 1 This is a flowchart illustrating the overall training and reasoning process of the method described in this invention.

[0025] Figure 2 This is a schematic diagram of the structure of the model described in this invention under DINO perception guidance and the rescaling CFG inference module.

[0026] Figure 3It measures the performance of tongue diagnosis in diabetes patients under different ratios of synthetic tongue images and real tongue images.

[0027] Figure 4 This is a sample tongue image generated by the tongue image data enhancement generation method described in this invention. Detailed Implementation

[0028] The specific embodiments of the present invention will be described below with reference to the accompanying drawings and examples. However, the following examples are only used to illustrate the present invention in detail and do not limit the scope of the present invention in any way.

[0029] Example 1: As Figure 1 As shown, it includes the following steps:

[0030] S1, Data Preparation and Preprocessing. The tongue image data in this implementation example comes from research related to diabetes detection based on tongue images (Kaggle platform diabetes tongue image dataset (https: / / www.kaggle.com / datasets / thngdngvn / biohit-tongue-image-dataset)). This dataset contains approximately 300 real clinical tongue images, divided into three diagnostic categories: healthy, pre-diabetes, and diabetes. Although these images have undergone pre-segmentation, the proportion of the tongue body in the images varies significantly, and the original sizes are inconsistent. This step first resizes all input tongue images to 512×512 pixels; then, it is fed into the encoder of a pre-trained variational autoencoder (VAE) to compress it into a 64×64×4 latent representation space, outputting the tongue image target latent vector x1.

[0031] S2, Model Construction and Training for Latent Space Flow Matching. Based on the flow matching objective, a linear probability path x is established connecting standard Gaussian white noise (x0) and the latent vector (x1) of the target tongue image sample. t =(1−t)x0+tx1. A UNet backbone network based on an adaptive group normalization (AdaGN) microstructure is constructed, where the time step (t) and class label (e.g., y representing different stages of diabetes) are jointly embedded and injected into each residual block of the UNet via AdaGN. The AdamW optimizer (maximum learning rate 2e-4) is used, combined with a Cosine Annealing learning rate decay strategy, with a batch size of 8 and a total training duration of 500 epochs. The mean squared error between the predicted tongue velocity field and the actual tongue velocity field is calculated as the model training loss, and backpropagation is used to update the UNet network parameters.

[0032] S3, DINO-aware guidance intervention. After 50 epochs of network training warm-up, and for specific stages (e.g., current time step t > 0.5), a DINO semantic constraint mechanism is introduced. This utilizes the current time step (t) and the velocity field (v) predicted by UNet. θ (x t The final hidden state is directly extrapolated from t,y). =x t +(1−t)v θ (x t The final tongue image latent vector is input into the frozen VAE decoder to restore it to a pixel image. This pixel image is then input along with the real tongue image into the frozen DINOv3 visual large model, where the deep semantic feature maps of both are extracted. The mean squared error loss (Lij) between the two feature vectors is calculated. DINO Multiply by the set weight value (0.05) and add it to the total model training loss L. Total In this approach, perception guidance aims to force the model to maintain the texture and complex anatomical information of the tongue image in different states throughout the generated trajectory.

[0033] S4 utilizes rescaled CFG image inference and tongue image data augmentation. Tongue images are generated based on a trained latent space flow matching model, starting with pure Gaussian noise and progressively denoising using ordinary differential equations. In each integration step, the model simultaneously calculates the conditional velocity field v. cond and unconditional velocity field v uncond v is calculated according to the conventional CFG formula. cfg =v uncond +ω⋅(v cond -v uncond To avoid contrast distortion and color fading due to an excessively large guide scale ω, v was calculated. cond and v cfg Standard deviation, v cfg Scaled according to variance, then factored through a dynamic interpolation block. Feature fusion is performed. The latent vector state is updated using the Euler method, and finally the UNet network predicts the output tongue image velocity field. The latent vector of the tongue image is generated by the frozen pre-trained VAE decoder and generates a pixel-space tongue image.

[0034] To verify the effectiveness of the generated synthetic tongue image data for downstream tongue diagnosis classification, this example uses 70% of the real tongue images in the diabetes dataset as the training set and the remaining 30% as a fixed test set. The ResNet-50 classification model is used as the tongue diagnosis classification network. The main experimental results are as follows:

[0035] (1) The effectiveness of generating tongue images for tongue diagnosis in diabetes.

[0036] like Figure 3 As shown, by adding different numbers of generated tongue images to 70% of the diabetes training set (defined as Fakeratio = number of synthetic tongue image samples / number of real tongue image samples), it can be seen that as the proportion of generated tongue image samples increases, the diabetes tongue diagnosis classification indicators (including accuracy, AUC, and F1 score) all show a steady upward trend. This indicates that the synthetic tongue image data generated by this invention can effectively supplement the feature distribution of real clinical tongue image data and improve the performance of tongue diagnosis.

[0037] (2) Effectiveness analysis of key modules

[0038] To systematically verify the independent and collaborative contributions of each key module in this invention, ablation experiments were conducted on a diabetes dataset, as shown in Table 1 below. The results show that the baseline flow matching model (Model 1) using only simple additional conditions achieved an accuracy of only 71.43%. Replacing conditional injection with AdaGN (Model 2) improved feature alignment, immediately increasing the accuracy to 75.32%. Further integration of DINO-aware guidance (Model 3) effectively prevented the loss of pathological textures, increasing the accuracy to 76.14%. Finally, the addition of the Rescaled CFG algorithm during the inference period effectively addressed color distortion and ensured variance matching, resulting in a significant leap in classification accuracy to 80.52% and an AUC of 90.29%. This verifies the superior performance of the proposed multi-module collaborative architecture in the field of TCM tongue image generation. Examples of tongue images generated under different key module models are shown below. Figure 4 As shown, the method of the present invention can also be seen visually. Figure 4 The last column generates a more realistic tongue image.

[0039] Table 1. Effectiveness Analysis of Key Modules

[0040] Model 1 Additive None Standard CFG 71.43 81.04 Model 2 AdaGN None Standard CFG 75.32 84.00 Model 3 AdaGN DINO Standard CFG 76.14 86.34 Method of the present invention AdaGN DINO Rescaled CFG 80.52 90.29

[0041] The present invention has been described in detail above with reference to the embodiments; however, those skilled in the art will understand that, without departing from the concept of the present invention, the specific parameters in the above embodiments can be changed, or the relevant components and structures can be equivalently substituted, thereby forming multiple specific embodiments, all of which are common variations of the present invention, and will not be described in detail one by one.

Claims

1. A method for enhancing and generating tongue image data based on DINO perception guidance, characterized in that, Includes the following steps: S1. Obtain real tongue image data labeled with different tag categories, and after preprocessing, use a pre-trained variational autoencoder (VAE) to map it to a low-dimensional latent space to obtain the target tongue image latent vector; S2, construct a time-dependent velocity field network with UNet architecture as the backbone and Adaptive Group Normalization (AdaGN) modulation. Use the flow matching mean square error loss as the basic loss function to train the network and learn the transformation trajectory from Gaussian white noise to the target tongue image latent vector. S3. In the model training phase, a perceptual guide based on the unlabeled distillation model DINO is introduced. When the model training rounds and the flow path time step parameters are greater than the preset threshold, the semantic features of the tongue image generated by the UNet prediction latent vector and the real tongue image are extracted using the frozen DINO model and the mean square error is calculated. The flow matching loss and the DINO perceptual loss are weighted and summed to obtain the total loss. The velocity field prediction network UNet is then updated through backpropagation. S4. In the inference sampling stage, a rescaled classifier-free guidance (CFG) strategy is used to synthesize the tongue image latent vector, which is then input into the pre-trained VAE decoder and mapped back to the image pixel space to generate the final tongue image of a specific label category.

2. The method for enhancing and generating tongue image data based on DINO perception guidance according to claim 1, characterized in that: In step S1: The real tongue images with different labels are uniformly adjusted to a preset size, and the tongue image latent vector is obtained by using a pre-trained variational autoencoder (VAE) to compress them into a low-dimensional latent space.

3. The method for enhancing and generating tongue image data based on DINO perception guidance according to claim 1, characterized in that: In step S2: a velocity field prediction network based on the UNet architecture is constructed. The time step and tongue image category label are jointly embedded into the residual block of the network through Adaptive Group Normalization (AdaGN). The training objective of the network is to minimize the mean square error between the predicted velocity field and the real tongue image latent vector velocity field.

4. The method for enhancing and generating tongue image data based on DINO perception guidance according to claim 1, characterized in that: In step S3: To prevent the loss of subtle tongue image features in the highly compressed latent space, DINO perceptual guidance is activated when the number of training epochs and the time step are greater than a given threshold (e.g., training epochs > 50 and time step > 0.5). First, the predicted final tongue image latent vector is extrapolated based on the intermediate latent vector at the current time step and the UNet predicted latent vector. Then, the final tongue image latent vector is decoded into the pixel space using the frozen VAE decoder to obtain the predicted tongue image image. Finally, the predicted tongue image image and the original real tongue image image are input into the frozen pre-trained DINO network to extract semantic features. The mean square error between the two features is calculated as the perceptual loss guide, and the weighted sum with the flow matching mean square error loss is used as the overall optimization objective for subsequent network training.

5. The method for enhancing and generating tongue image data based on DINO perception guidance according to claim 1, characterized in that: In step S4: During the inference phase, starting from the initial time step (t=0) with Gaussian white noise, the trained tongue image velocity field Unet prediction network is used to calculate the predicted values ​​of the conditional velocity field and the unconditional velocity field at each time step (t), thereby deriving the standard CFG conditional velocity field. Subsequently, the standard deviations of the corresponding conditional and unconditional velocity fields are calculated, and the standard CFG conditional velocity field value is rescaled using the ratio of the two. The standard CFG conditional velocity field and the rescaled standard CFG velocity field are then fused by interpolation to obtain the final prediction of the tongue image velocity field at that time step (t). Update the next time step (t+) using Euler's method. The tongue image latent vector is calculated until the time step is 1 (t=1); finally, the predicted tongue image latent vector is input into the pre-trained VAE decoder to generate a high-quality tongue image of a specific label category.

Images