A tongue disease risk prediction method and system based on prompt mutual learning
By employing a cue-based mutual learning approach, and leveraging the mutual learning between the visual language teacher model and the student model, the performance of the CLIP model in tongue image analysis was addressed due to discrepancies in cue design and data distribution. This approach enabled efficient and accurate prediction of tongue image disease risks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2025-09-08
- Publication Date
- 2026-06-26
AI Technical Summary
In existing technologies, the CLIP model's performance in tongue image analysis is greatly affected by the cue design, has high inference costs, and the distribution of pre-training data differs greatly from that of professional tongue image images, resulting in low accuracy in disease risk prediction.
We adopt a cue-based mutual learning approach, which involves mutual learning between the visual language teacher model and the student model. By combining KL divergence loss and consistency loss, we design learnable cuees to reduce computation and improve model robustness. We then use pre-stored text feature vectors to predict disease risk.
Without affecting the generalization ability of the visual language teacher model, it significantly improves the accuracy and efficiency of tongue disease risk prediction, reduces computational overhead, and is suitable for resource-constrained environments.
Smart Images
Figure CN121171582B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to a method and system for predicting tongue disease risk based on cue-based mutual learning. Background Technology
[0002] Tongue diagnosis is a core component of traditional Chinese medicine's "inspection" method. The tongue's appearance directly reflects the state of the body's internal organs, qi, blood, and body fluids, as well as the nature of pathogenic factors, possessing diagnostic value in "inferring internal conditions from external manifestations." However, traditional tongue diagnosis relies heavily on the physician's subjective experience, resulting in significant uncertainty and randomness.
[0003] With the development of technology, tongue image analysis is gradually becoming digital and objective. Modern optical equipment (such as multispectral cameras) is used to collect tongue images, and quantitative analysis of color and texture features is performed based on color space models (such as RGB, LAB, HSV), combined with classifiers to predict disease risk. However, this method is sensitive to the environment, has high equipment costs, and lacks unified tongue image benchmark parameters, resulting in poor comparability of results.
[0004] In recent years, deep learning has been introduced into tongue image analysis tasks. Mainstream models such as ResNet, ViT, and MLP have seen initial applications in this field, but significant limitations still exist:
[0005] ResNet excels at extracting local features (such as tongue coating texture), but it is insufficient in modeling global structural relationships and is easily affected by changes in lighting and color. ViT, while possessing global modeling capabilities, relies on a large amount of data, and its block processing can easily destroy subtle local features, resulting in high computational complexity. MLP requires image flattening, which destroys spatial structural information, has limited feature representation capabilities, and leads to low diagnostic accuracy.
[0006] The multimodal pre-trained model CLIP aligns image and text features through contrastive learning, exhibiting strong generalization ability and robustness to differences in lighting and equipment, providing a new approach to tongue image analysis. However, when directly applying CLIP, the pre-training data differs significantly from the distribution of tongue images in Traditional Chinese Medicine, making it difficult to capture subtle professional features. Secondly, the model's performance is highly dependent on the design of text prompts, requiring professional knowledge support. Furthermore, the model has a large number of parameters and high inference costs, making it difficult to deploy in resource-constrained environments. Summary of the Invention
[0007] In view of this, the present invention provides a method and system for predicting tongue disease risk based on cue mutual learning, aiming to at least partially solve the problems in the prior art, such as the CLIP model performance being greatly affected by cue design, high model inference cost, and low accuracy in predicting disease risk caused by large differences in the distribution of pre-training data and professional tongue images.
[0008] To achieve the above objectives, the present invention adopts the following technical solution:
[0009] First, this application discloses a method for predicting the risk of tongue-shaped diseases based on prompting and mutual learning, the steps of which include:
[0010] S1. Based on the tongue image dataset, a visual language teacher model is trained through cue learning and consistency loss, and the text feature vector generated by the visual language teacher model is saved.
[0011] S2. Initialize at least two student models to learn from each other, and introduce KL divergence loss with the visual language teacher model during mutual learning;
[0012] S3. Use any student model to extract the image feature vector of the tongue image to be predicted, multiply it with the text feature vector generated by the saved visual language teacher model, and output the disease risk prediction result.
[0013] Preferably, the cue learning includes visual cues and text cues;
[0014] The visual cues are obtained by weighted fusion of cues generated in multiple training cycles;
[0015] The text prompts are obtained by averaging the text feature vectors generated from different text prompt templates.
[0016] Preferably, the consistency loss includes:
[0017] The consistency between the features generated by cue learning and the output features of the original visual language teacher model, and the consistency between the prediction results obtained by cue learning and the prediction results of the original visual language teacher model.
[0018] Preferably, when pre-training the visual language teacher model, a cross-entropy loss is introduced on top of the consistency loss.
[0019] Preferably, both student models contain only an image encoder branch and are trained using learnable visual cues.
[0020] As a preferred approach, mutual learning between the two student models includes:
[0021] The student model dynamically selects roles based on the predicted entropy of the samples.
[0022] For models selected as student roles, the loss function guides them to move closer to the predictions of teacher role models.
[0023] For models selected as teachers, a reverse knowledge distillation strategy is used to guide predictions away from models that are designed for student roles.
[0024] Preferably, the model with lower prediction entropy is selected as the teacher role, and the model with higher prediction entropy is selected as the student role; the formula for calculating the prediction entropy is:
[0025]
[0026] In the formula, En(p) represents the predicted entropy value, p represents the predicted probability distribution output by the student model, K represents the total number of disease risk categories, and k represents the index of the disease risk category.
[0027] Second, this application discloses a tongue disease risk prediction system based on prompting and mutual learning, including:
[0028] The visual language teacher model training module is used to train a visual language teacher model based on a tongue image dataset through cue learning and consistency loss, and to save the text feature vectors generated by the visual language teacher model.
[0029] The student model training module is used to initialize at least two student models for mutual learning, and the mutual learning introduces the KL divergence loss with the visual language teacher model.
[0030] The disease risk prediction module is used to extract the image feature vector of the tongue image to be predicted using any student model, multiply it with the text feature vector generated by the saved visual language teacher model, and output the disease risk prediction result.
[0031] Third, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the tongue image disease risk prediction method based on prompting mutual learning as described above.
[0032] And a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the tongue image disease risk prediction method based on cue mutual learning as described in any of the above.
[0033] As can be seen from the above technical solutions, the tongue image disease risk prediction method and system disclosed in this invention, based on prompt-based mutual learning, has the following advantages compared with the prior art:
[0034] (1) This invention uses self-prompting regularization to train the visual language teacher model. While learning specific tongue image classification task representations, it retains the generalized visual language teacher model knowledge, thereby maximizing the performance of downstream tongue image classification tasks without affecting the generalization of the visual language teacher model.
[0035] (2) Design learnable prompts, transfer the knowledge of the larger visual language teacher model to the student model through prompt-driven imitation, and pre-store the text feature vectors generated by the visual language teacher model for the student model's reasoning, which greatly reduces the computational load of student model training and reasoning and improves the reasoning efficiency of student model.
[0036] (3) The role selection strategy and reverse knowledge distillation strategy proposed in this invention dynamically set the relative teacher model and student model according to each sample in two small student models, guiding the two small student models to learn more effective knowledge from each other, which can effectively improve the robustness of the model. Attached Figure Description
[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0038] Figure 1 This is a flowchart of the tongue image disease risk prediction method based on prompting mutual learning of the present invention;
[0039] Figure 2 This is a schematic diagram of the regularized pre-trained CLIP teacher model based on prompting learning according to the present invention;
[0040] Figure 3 This is a schematic diagram illustrating the training of the CLIP student model using the distillation framework based on prompting mutual learning according to the present invention.
[0041] Figure 4 This is a schematic diagram of the role selection strategy of the present invention. Detailed Implementation
[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0043] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0044] This invention discloses a method and system for predicting tongue disease risk based on cue-based mutual learning. It is used to effectively apply the visual language teacher model to downstream tongue classification tasks through cue-based learning, while retaining the generalization ability of the original visual language teacher model. Furthermore, by pre-storing text feature vectors and knowledge distillation, it reduces the amount of computation and model parameters, thereby improving the accuracy and efficiency of tongue disease risk prediction.
[0045] In some embodiments, tongue image disease risk prediction methods based on cue-based mutual learning, such as Figure 1 The steps include,
[0046] S1. Based on the tongue image dataset, a visual language teacher model is trained through cue learning and consistency loss, and the text feature vector generated by the visual language teacher model is saved.
[0047] S2. Initialize at least two student models to learn from each other, and introduce KL divergence loss with the visual language teacher model during mutual learning;
[0048] S3. Use any student model to extract the image feature vector of the tongue image to be predicted, multiply it with the text feature vector generated by the saved visual language teacher model, and output the disease risk prediction result.
[0049] In this embodiment, the visual language teacher model is preferably CLIP, and it will be used as an example for explanation below.
[0050] In some implementations, in step S1,
[0051] First, the original tongue image is obtained. Then, the tongue image is extracted using a tongue segmentation model to construct a tongue image dataset. This dataset is used to guide the CLIP teacher model through prompting learning, thereby improving its applicability in the medical field and enabling it to adapt to downstream tongue image classification tasks, thus enhancing the performance of the CLIP teacher model on downstream tasks.
[0052] Optionally, this application uses a pre-trained object detection model to detect and extract the tongue region from the original image, and then crops it to obtain a standardized-size tongue image. Specifically, the original tongue image and its corresponding disease label are obtained. For the original tongue image, a pre-trained YOLOv8 tongue object detection model is used to accurately locate and extract the tongue region, which is then cropped and standardized to a uniform 512×512 pixel image.
[0053] To alleviate data scarcity and sample class imbalance, image enhancement strategies were further applied to the training set samples. Specifically, the images were horizontally flipped with a probability of 0.5, and the brightness, contrast, saturation, and hue of the images were randomly adjusted to simulate tongue images under different lighting conditions. The images were also randomly rotated by 15 degrees.
[0054] Furthermore, a visual language teacher model is trained using cue-based learning and consistency loss.
[0055] This invention introduces a cue self-regularization framework in this step to design learnable cues to guide the CLIP teacher model to adapt to downstream medical tongue image classification tasks using a constructed tongue image dataset. Specifically, to avoid the CLIP teacher model losing its original generalization ability due to overfitting of cues to the downstream tongue image data distribution, this invention introduces a cue self-regularization framework, which mainly includes three parts: adjusting the cue representation by maximizing the consistency with the original frozen CLIP teacher model; adjusting through self-integration of cues during training to encode the complementary advantages of cues; and adjusting through text diversity to mitigate the sample diversity imbalance in the visual branch.
[0056] In one specific embodiment, such as Figure 2 As shown:
[0057] The adjustment for mutual consistency is mainly divided into two parts: visual and textual. The visual branch takes a cropped and standardized tongue image x as input. image Learnable image cues are added, and the image is processed through an image encoder f(x) to generate image feature vectors.
[0058] Text branches use a series of text templates x text Image categories are stitched together to enrich the semantic features of the text, learnable text cues are added, and the text is then processed by a text encoder g(x) to generate a text feature vector.
[0059] Finally, by using image features and text features The final prediction logic is obtained by using vector dot product.
[0060] Simultaneously, for a given input sample and its corresponding text label, the visual features obtained by the frozen CLIP model are obtained. Text features obtained from the frozen CLIP model
[0061] Therefore, L1 loss is used to impose feature-level consistency constraints on the visual and textual features of the prompt, as shown below.
[0062]
[0063] In the formula, Image feature vectors generated for the trained CLIP teacher model image encoder. The image feature vector generated by the frozen original CLIP teacher model image encoder, where d is the feature vector dimension, i represents the index of the feature vector, and L... SCL-image for and L1 loss between The text feature vectors generated for the text encoder of the trained CLIP teacher model. L is the text feature vector generated by the frozen original CLIP teacher model text encoder. SCL-text for and L1 loss between.
[0064] Furthermore, a self-regularization constraint is added to maximize the alignment between frozen features and cue features. A consistency constraint is imposed on the prediction logic level by minimizing the KL divergence, as shown below:
[0065]
[0066] In the formula, express and cosine similarity, express and cosine similarity, D KL This indicates the calculation of the KL divergence, where L... SCL-logits This represents the KL divergence between the trained CLIP teacher model prediction logic and the frozen original CLIP teacher model prediction logic.
[0067] Finally, the self-consistency loss used is as follows:
[0068] L SCL =λ1L SCL-image +λ2L SCL-text +L SCL-logits
[0069] In the formula, L SCL-image For image feature level loss, L SCL-text For text feature level loss, L SCL-logits To predict the logical level loss, λ1 is the weight of the image feature level loss, λ2 is the weight of the text feature level loss, and L SCL This is the final self-consistency loss value.
[0070] In this implementation, the image feature-level consistency constraint weight λ1 is set to 10, the text feature-level consistency constraint weight λ2 is set to 25, the learning rate is set to 0.0025, the stochastic gradient descent (SGD) optimizer and cosine learning rate scheduler are used, the number of training epochs is set to 20, and the batch size is set to 16.
[0071] To further optimize the above technical solution, when pre-training the CLIP teacher model, a cross-entropy loss is introduced on top of the consistency loss, i.e.
[0072] L final =L CE +L SCL
[0073] In the formula, L CE For the cross-entropy loss between the predicted and true values of the trained CLIP teacher model, L SCL For the aforementioned self-consistency loss, L final This is the final training loss value.
[0074] Furthermore, in the above scheme, the tongue image x image The category is x cls On the visual branch, for image x image Add learnable visual cues to the text branch, setting the category x. cls With text templates (such as "a photo of a{x") cls Combine}”) and add learnable text prompts;
[0075] For visual cues, this application obtains integrated visual cues by weighted fusion of cues generated over multiple training epochs; when the total number of iterations is E, the cues for each epoch are... By using the Gaussian distribution w t ~N(μ,σ 2 The optimal weight value w is obtained by sampling in the middle. t , where μ and σ 2 It is a hyperparameter, and The integrated prompt calculation is as follows:
[0076]
[0077] In the formula, E is the total number of training iterations, t is the current round, and w t Let {P} be the optimal weight value for round t, and P be the hint for each round. AP This is the aggregated suggestion.
[0078] For text prompts, this application obtains an integrated text prompt by averaging the text feature vectors generated from different text prompt templates, which helps alleviate the sample diversity imbalance in the visual branch. This embodiment uses a pool of text prompt templates containing N enhanced forms for each category. The prompt text is enhanced by integrating multiple prompt templates, and the text features of the pre-trained CLIP teacher model are obtained as follows:
[0079]
[0080] In the formula, N represents the number of text prompts in the text prompt template pool. The text feature vector generated for the nth text prompt template. This is the text feature vector after aggregating the feature vectors of N prompt template texts.
[0081] This invention designs a learnable cue training CLIP teacher model through the above three standard methods, which can maximize the performance of downstream tongue image classification tasks without affecting CLIP generalization.
[0082] In a preferred embodiment, in order to significantly reduce the amount of computation and ensure high-quality text feature vectors, the present invention pre-stores the text feature vectors generated by the CLIP teacher model after training the large CLIP teacher model, for use in the subsequent training and inference of the CLIP student model.
[0083] In some implementations, step S2 includes: initializing at least two student models to learn from each other, wherein the mutual learning incorporates a KL divergence loss with the visual language teacher model;
[0084] Preferably, the student model in this embodiment is the CLIP student model. Since it reuses the text feature vectors saved in the pre-training teacher stage, the CLIP student model in this application only contains the image encoder branch and is trained using learnable visual cues.
[0085] In this embodiment, learnable prompts are designed to transfer the knowledge reasoning of the pre-trained large CLIP teacher model to the CLIP student model through prompt-driven imitation.
[0086] Specifically, such as Figure 3 As shown, two student models are initialized, and the two student models are distilled using a pre-trained large CLIP teacher model. During training, on the one hand, KL divergence is used to align the logic of the teacher and student models, encouraging the student image encoder to generate a probability distribution similar to the teacher's through learnable cues. On the other hand, for the two student models, this invention proposes a role selection strategy, such as... Figure 4 As shown, the relative prediction entropy of the two student models for each data sample is dynamically set to the predictions of the teacher model and the student model. For the student model, it is guided to predict predictions that are close to those of the teacher model. For the teacher model, this invention proposes a reverse knowledge distillation strategy to guide it to predict predictions that are far from those of the student model.
[0087] In one specific embodiment, the model with lower prediction entropy is used as the teacher model, and the model with higher prediction entropy is used as the student model. For sample x, the formula for calculating prediction entropy is as follows.
[0088]
[0089] In the formula, En(p) represents the predicted entropy value, p represents the predicted probability distribution output by the student model, K represents the total number of disease risk categories, and k represents the index of the disease risk category.
[0090] In each iteration, the target samples are sampled from a batch of samples B. Based on the role selection strategy criteria, the target samples can be divided into two groups, B1 and B2, as follows.
[0091] B1={x i |En(p i,1 ) <En(p i,2 )}
[0092] B2={x i |En(p i,1 )>En(p i,2 )}
[0093] Where, p i,1 =f(x; θ) stu1 ) is the prediction logic of student model 1, p i,2 =f(x; θ) stu2 ) is the prediction logic of student model 2, En(p i,1 En(p) is the prediction entropy of student model 1. i,2 ) is the prediction entropy of student model 2, x i Let represent the i-th sample. According to the rules of the role selection strategy,
[0094] For set B1, student model 1 is selected as the teacher model, and knowledge needs to be distilled from student model 1 to student model 2. The losses for student models 1 and 2 are:
[0095] Student Model 1:
[0096] Student Model 2:
[0097] In the formula, Let B1 be the L2 loss for the prediction logic of student model 1 and the prediction logic of student model 2, and L be the sample set. rkd (B1) represents the inverse knowledge distillation loss of student model 1, L tkd (B1) represents the positive knowledge distillation loss of student model 2.
[0098] For set B2, student model 2 is selected as the teacher model. Knowledge needs to be distilled from student model 2 to student model 1. The losses for student models 1 and 2 are:
[0099] Student Model 1:
[0100] Student Model 2:
[0101] Finally, for both Student Model 1 and Student Model 2, a KL divergence of a pre-trained large CLIP teacher model is added to all target sample data. This allows the student models to align with the predictions of the large CLIP teacher model through cue imitation. Therefore, the final loss for Student Model 1 and Student Model 2 is:
[0102]
[0103] Among them, D KL (logits tea ,logits s1 ) represents the KL divergence between the large CLIP teacher model and student model 1, and D KL (logits tea ,logits s2 α1 represents the KL divergence between the large CLIP teacher model and student model 2, α2 represents the weight of the forward knowledge distillation loss, α3 represents the weight of the backward knowledge distillation loss, and α4 represents the weight of the KL divergence.
[0104] In a preferred embodiment, during the early training phase (the first 10 training epochs), the error loss D between the prediction logic of the student model and the large teacher model is calculated using KL divergence. KL (logits tea ,logits s This allows student models to learn knowledge reasoning from large teacher models. In the later stages of training (the last 10 training rounds), additional knowledge learning between the two student models is added.
[0105] In this implementation, during the distillation phase of mutual learning, the learning rate is set to 0.005, a stochastic gradient descent (SGD) optimizer and a cosine learning rate scheduler are used, the number of training epochs is set to 20, the batch size is set to 16, and the weight ratio of KL divergence and knowledge distillation loss and reverse knowledge distillation loss in the later stage of training is set to 6:3:1.
[0106] In some implementations, step S3 involves using any CLIP student model to extract the image feature vector of the tongue image to be predicted, and then performing inference with the saved text feature vector generated by the CLIP teacher model.
[0107] During the inference phase, for the test sample data, the image encoder of the CLIP student model is used to process the sample data to obtain student image features. The student image features are then multiplied by the pre-stored teacher text features to obtain the final prediction logic, thus completing the task of predicting disease risk by tongue appearance.
[0108] Specifically, for the test sample data, the image encoder of the CLIP student model is used. Process image x to obtain student image features Pre-stored teacher text features W∈R N×d Then the output prediction logic q t =u s ×W T ∈R N W T This is the transpose of W.
[0109] This invention first utilizes a self-regularization framework to train a large teacher model, ensuring high-quality capture and strong generalization ability of subtle features of TCM tongue diagnosis. Subsequently, through an innovative role selection and backdistillation mechanism, it drives a lightweight student model to learn efficiently in a competitive environment. Ultimately, only a single student model is needed to achieve prediction accuracy comparable to that of a large model, while significantly reducing computational overhead and deployment threshold, providing an efficient and reliable solution for the intelligentization and standardization of TCM tongue diagnosis.
[0110] The prediction method of this invention can be executed by a computer program to predict disease risk based on a trained tongue image dataset. It can effectively extract tongue image features and achieve a high prediction accuracy.
[0111] In one embodiment, the hardware implementation includes:
[0112] 1. Establish a tongue image disease risk prediction system based on prompt-based mutual learning, including:
[0113] The visual language teacher model training module is used to train a visual language teacher model based on a tongue image dataset through cue learning and consistency loss, and to save the text feature vectors generated by the visual language teacher model.
[0114] The student model training module is used to initialize at least two student models for mutual learning, and the mutual learning introduces the KL divergence loss with the visual language teacher model.
[0115] The disease risk prediction module is used to extract the image feature vector of the tongue image to be predicted using any student model, multiply it with the text feature vector generated by the saved visual language teacher model, and output the disease risk prediction result.
[0116] 2. An electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the tongue image disease risk prediction method based on prompting mutual learning as described above.
[0117] 3. A computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the tongue image disease risk prediction method based on prompting mutual learning as described in any one of the above.
[0118] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0119] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for predicting tongue disease risk based on cue-based mutual learning, characterized in that, include: S1. Based on the tongue image dataset, a visual language teacher model is trained through cue learning and consistency loss, and the text feature vector generated by the visual language teacher model is saved. S2. Initialize at least two student models to learn from each other, and introduce KL divergence loss with the visual language teacher model during mutual learning; The two student models learn from each other, including: The student model dynamically selects roles based on the predicted entropy of the samples. For models selected as student roles, the loss function guides them to move closer to the predictions of teacher role models. For models selected as teachers, a reverse knowledge distillation strategy is used to guide predictions away from models that are designed for student roles. In this model, the model with lower prediction entropy is selected as the teacher role, and the model with higher prediction entropy is selected as the student role; the formula for calculating the prediction entropy is: In the formula, This represents the predicted entropy value. This represents the predicted probability distribution output by the student model. This represents the total number of disease risk categories. An index representing disease risk categories; S3. Use any student model to extract the image feature vector of the tongue image to be predicted, multiply it with the text feature vector generated by the saved visual language teacher model, and output the disease risk prediction result.
2. The tongue image disease risk prediction method based on prompting mutual learning according to claim 1, characterized in that, The cue learning includes visual cues and text cues; The visual cues are obtained by weighted fusion of cues generated in multiple training cycles; The text prompts are obtained by averaging the text feature vectors generated from different text prompt templates.
3. The tongue image disease risk prediction method based on prompting mutual learning according to claim 1, characterized in that, The consistency loss includes: The consistency between the features generated by cue learning and the output features of the original visual language teacher model, and the consistency between the prediction results obtained by cue learning and the prediction results of the original visual language teacher model.
4. The tongue image disease risk prediction method based on prompting mutual learning according to claim 3, characterized in that, When pre-training the visual language teacher model, a cross-entropy loss is introduced on top of the consistency loss.
5. The tongue image disease risk prediction method based on prompting mutual learning according to claim 1, characterized in that, Both student models contain only an image encoder branch and are trained using learnable visual cues.
6. A tongue image disease risk prediction system based on prompting and mutual learning, characterized in that, The tongue image disease risk prediction method based on cue-based mutual learning as described in any one of claims 1 to 5 includes: The visual language teacher model training module is used to train a visual language teacher model based on a tongue image dataset through cue learning and consistency loss, and to save the text feature vectors generated by the visual language teacher model. The student model training module is used to initialize at least two student models for mutual learning, and the mutual learning introduces the KL divergence loss with the visual language teacher model. The disease risk prediction module is used to extract the image feature vector of the tongue image to be predicted using any student model, multiply it with the text feature vector generated by the saved visual language teacher model, and output the disease risk prediction result.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the tongue image disease risk prediction method based on prompting mutual learning as described in any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the tongue image disease risk prediction method based on prompting mutual learning as described in any one of claims 1 to 5.