Tongue picture recognition auxiliary diagnosis method and device based on multiple modes and terminal equipment

Abstract

The invention belongs to the field of intelligent tongue diagnosis, and relates to a multi-modality-based tongue picture recognition auxiliary diagnosis method, which comprises the following steps: acquiring multi-modality tongue picture data, and inputting the multi-modality tongue picture data into a trained tongue picture auxiliary diagnosis model to obtain a diagnosis result; the training process of the model comprises the following steps: acquiring a pre-trained CLIP model, and injecting the LoRA and MaPLe modules into the CLIP model to obtain a tongue picture auxiliary diagnosis model; acquiring a multi-modal tongue picture data set, including image and category text prompts; inputting the image and the category text prompt into a tongue picture auxiliary diagnosis model to obtain visual features and text features; calculating the similarity between the visual features and the text features to obtain a logarithm probability vector; calculating a loss function value according to the logarithm probability vector, and updating parameters of the LoRA and MaPLe modules until model training is completed; loRA and MaPLe are injected into the CLIP model, consumption of computing resources is reduced, and the problem of disastrous forgetting when the model is finely adjusted on a small sample is avoided.

Description

Technical Field

[0001] This invention belongs to the field of intelligent tongue diagnosis, and relates to a multimodal tongue image recognition-assisted diagnosis method, device, and terminal equipment. Background Technology

[0002] The research process of automatic tongue image analysis in traditional Chinese medicine has mainly gone through two stages: the traditional method based on manual feature extraction and shallow machine learning, and the end-to-end intelligent analysis method based on deep learning.

[0003] In the traditional stage based on manual feature extraction and shallow machine learning, existing techniques typically preprocess tongue images and then extract manually designed features (such as color histograms, color sets, texture structuring element descriptors, color co-occurrence matrices, image moments, etc.). After obtaining the features, traditional machine learning classifiers are used for recognition. For example, existing techniques include methods that use block truncation encoding to extract color features; schemes based on the AdaBoost cascaded classification framework to improve the accuracy of tongue color classification; techniques that extract color features in RGB space and then use support vector machines (SVM) or K-nearest neighbors (KNN) as classifiers; and studies that use dominant hue histograms combined with multi-label learning algorithms to classify tongue body and tongue coating. In addition, color space selection based on Fisher ratio is also a common method in this stage. This stage of methods heavily relies on manually designed features, which are often particularly sensitive to changes in lighting and differences in acquisition equipment, resulting in generally poor model robustness and classification accuracy in complex scenarios that is difficult to meet the needs of clinical auxiliary diagnosis.

[0004] In the stage of end-to-end intelligent analysis methods based on deep learning, with the development of deep learning, convolutional neural networks (CNNs) have been widely used for the automatic extraction and classification of tongue image features. Existing technologies include schemes that use improved CaffeNet, ResNet, or custom CNN network structures to classify tongue images. To reduce the mutual interference between tongue color and tongue coating color, some techniques separate the entire tongue image into different regions and feed them into lightweight CNN networks for processing. Furthermore, some research has begun to attempt to combine image and text features, using multimodal fusion to improve classification performance. Although deep learning-based methods significantly outperform traditional methods, the following core bottlenecks still exist, leading to unsatisfactory classification results:

[0005] 1. Strong data dependence and high annotation costs: Deep neural network models have a large number of parameters and complex structures, requiring massive amounts of labeled samples for training to converge. However, the collection of high-quality tongue image data and the annotation by professional doctors are time-consuming and labor-intensive, resulting in an extreme scarcity of labeled data, which severely limits the upper limit of model performance.

[0006] 2. Insufficient generalization ability: Existing models are mostly trained on single, small-scale datasets (usually from the same device). When lighting or acquisition equipment changes, the difference in data distribution can lead to a significant drop in model performance.

[0007] 3. Insufficient utilization of single-modal information: Most existing intelligent tongue diagnosis research still focuses on a single image modality, failing to fully explore the rich semantic information contained in the label text (such as descriptive words like "thin white" and "yellow and greasy"). The model lacks an understanding of textual semantics, especially when distinguishing fine-grained tongue features such as "pale white" and "pale red," where relying solely on visual information can easily lead to confusion.

[0008] To address the aforementioned issues, particularly the "unimodal information bottleneck," pre-trained vision-language models (VLMs), such as CLIP (Contrastive Language-Image Pre-training), offer a novel approach. VLMs, through contrastive learning on billions of image-text pairs, achieve powerful cross-modal understanding and zero-shot generalization capabilities, giving them the potential to surpass unimodal models in information fusion and robustness. However, directly applying VLMs like CLIP to fine-grained tongue image classification tasks faces unique challenges:

[0009] 1. Domain Gap: CLIP models are pre-trained on general internet natural image data (such as animals, vehicles, etc.), while tongue images are highly specialized medical images. There are significant differences in data distribution between the two, and direct use is not effective.

[0010] 2. Fine-grained recognition misalignment: CLIP models tend to learn the global semantic features of images, while tongue image classification often relies on subtle local feature differentiation (for example, distinguishing between "white greasy coating" and "yellow greasy coating" mainly relies on subtle differences in color and texture). General models have difficulty achieving accurate perception of such fine-grained features.

[0011] 3. High cost of fine-tuning and catastrophic forgetting: Large models such as CLIP have a huge number of parameters (e.g., ViT-B / 16 architecture). Full parameter fine-tuning not only requires huge computing resources, but also is prone to overfitting on small sample datasets such as tongue images, causing the model to forget the general visual knowledge learned in the pre-training stage (i.e., "catastrophic forgetting").

[0012] In summary, existing technologies lack a tongue image recognition scheme that can effectively overcome the challenges of data scarcity, large cross-domain distribution differences, and difficulties in aligning fine-grained features. Summary of the Invention

[0013] To address the aforementioned problems in the prior art, this invention employs a multimodal tongue image recognition-assisted diagnosis method, comprising: acquiring multimodal tongue image data; inputting the multimodal tongue image data into a trained tongue image-assisted diagnosis model to obtain a diagnosis result; the training process of the tongue image-assisted diagnosis model includes:

[0014] S1. Obtain the pre-trained CLIP model, construct the LoRA module and the MapLe module, and inject the LoRA module and the MapLe module into the pre-trained CLIP model to obtain the tongue image auxiliary diagnosis model; among them, the LoRA module is a low-rank matrix adaptive module, and the MapLe module is a multimodal prompt word module.

[0015] S2. Obtain the multimodal tongue image dataset, which includes multiple tongue image images and category text prompt templates; the category text prompt templates include category text prompts for all categories of the tongue image images;

[0016] S3. Input each tongue image and the category text prompt for each category into the tongue image auxiliary diagnosis model to obtain the visual features of each tongue image and the text features of each category;

[0017] S4. Calculate the similarity between the visual features of each tongue image and the text features of each category to obtain the logits vector of each tongue image; calculate the loss function value based on the logits vector of all tongue images; where the logits vector is the log odds vector.

[0018] S5. Freeze the parameters of the pre-trained CLIP model in the tongue image auxiliary diagnosis model, update the parameters of the LoRA module and the MapLe module according to the loss function value, and obtain the trained tongue image auxiliary diagnosis model when the loss function value is minimized.

[0019] Beneficial effects:

[0020] 1. This invention integrates image information and textual semantic information from labels, greatly improving the reliability of model diagnosis. 2. This invention utilizes the powerful prior knowledge of the CLIP model and combines it with LoRA technology to freeze the CLIP backbone weights during training, training only a small number of parameters of the newly injected LoRA and MaPLe modules. This greatly reduces computational resource consumption and effectively avoids the "catastrophic forgetting" problem that occurs when the model is fine-tuned on small sample data. 3. This invention constructs a cross-modal bridge through an innovative MaPLe cross-modal coupling cue word mechanism, allowing gradients to propagate bidirectionally between cue words in two modal branches. This forces the semantic concepts learned by the model on the text side (such as "greasy moss") to find corresponding fine-grained features (such as specific textures or thicknesses) on the visual side, thereby achieving stronger feature space alignment and accurately solving the problem of fine-grained recognition misalignment. Attached Figure Description

[0021] Figure 1 A flowchart illustrating a multimodal tongue image recognition-based auxiliary diagnostic method provided in an embodiment of the present invention;

[0022] Figure 2 This is a structural diagram of the tongue image-assisted diagnostic model provided in an embodiment of the present invention;

[0023] Figure 3 This is a structural diagram of the encoder layer with injected LoRA provided in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a multimodal tongue image recognition-assisted diagnostic device provided in an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of a multimodal tongue image recognition-assisted diagnostic terminal provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] like Figure 1 As shown, this embodiment of the invention employs a multimodal tongue image recognition-assisted diagnosis method, comprising: acquiring multimodal tongue image data, inputting the multimodal tongue image data into a trained tongue image-assisted diagnosis model, and obtaining a diagnosis result; the training process of the tongue image-assisted diagnosis model includes:

[0028] S1. Obtain the pre-trained CLIP model, construct the LoRA module and the MaPLe module, and inject the LoRA module and the MaPLe module into the pre-trained CLIP model to obtain the tongue image auxiliary diagnosis model; among them, the LoRA (Low-Rank Adaptation) module is a low-rank matrix adaptive module, and the MaPLe (Multi-modal Prompt Learning) module is a multi-modal prompt word module;

[0029] like Figure 2 As shown, the pre-trained CLIP model includes: a visual encoder. and text encoder Visual encoder Including N(12) layers of visual encoders connected in series (Transformer layer), text encoder Includes N layers of text encoders connected in series. (Transformer layer);

[0030] In this embodiment, N is 12.

[0031] Injecting LoRA and MapLe modules into a pre-trained CLIP model includes:

[0032] S11, in each visual encoder layer and each text encoder layer Injecting a LoRA module into the self-attention module yields the visual encoder layer. and text encoder layer ;

[0033] like Figure 3 As shown, in the visual encoder layer Or text encoder layer Injecting LoRA modules into the self-attention module includes:

[0034]

[0035]

[0036]

[0037] Where x is the input feature of the self-attention module, , , These are the query matrix, key matrix, and value matrix injected into LoRA, respectively. These are the visual encoder layers. Or text encoder layer The projection matrix of the query matrix Q, key matrix K, and value matrix V in the self-attention module ( ), where d and k are the dimensions of the projection matrix. , ( Let be a low-rank decomposition matrix that can be trained using the LoRA module, and let its rank be... , This is the scaling factor.

[0038] Preferably, r=2, which is the optimal choice based on ablation experiments. This method introduces only a very small fraction of the total number of trainable parameters in CLIP, but during the training and inference phases, for example... , can be with It merges without introducing any additional computational delay.

[0039] S12, each visual encoder layer in the first J layer and text encoder layer Before that, add a MapLe module. .

[0040] Unlike traditional methods that only add prompts at the input layer, this invention injects learnable prompt words into both the first J-layer visual encoder and the text encoder layer, solving the misalignment problem of existing visual-language models in recognizing fine-grained features of the tongue image (such as tongue coating and cracks).

[0041] Text side: In the text encoder layer ( ), introducing dimensions as Learnable text prompts.

[0042] Visual end: at the visual encoder layer ( ) Introducing dimensions as Learnable visual cues.

[0043] S2. Obtain a multimodal tongue image dataset, which includes multiple tongue images. And category text hint templates; where the category text hint templates include category text hints for all categories of the tongue image, and k is the index of the image;

[0044] The TCMID-Tongue public dataset was used, containing 1472 tongue images, which were divided into 5 categories: Mirror-Approximated, Thin-white, White-Greasy, Yellow-Greasy, and Grey-Black.

[0045] In one embodiment, data augmentation of the dataset includes: for class imbalance problems (such as Grey-Black having only 252 images), randomly flipping, rotating (-45° to 45°), cropping, and scaling the few sample classes.

[0046] Category text prompt template ; where, category text hint for category j ;in, Let J be the j-th category of the tongue image (e.g., "thin white coating"), and C be the number of categories.

[0047] S3. Input each tongue image and the category text prompt for each category into the tongue image auxiliary diagnosis model to obtain the visual features of each tongue image and the text features of each category;

[0048] For each image The processing steps of the tongue image-assisted diagnostic model, including the category cue words for each category j, include:

[0049] S31. Tongue image Perform block embedding to obtain the image embedding representation. ; Category text hints Perform word embedding to obtain the category text embedding representation. ;

[0050] S32, According to the MapLe module Get the initial text prompt and initial visual cues Embedding images into representations and initial visual cues The data is stitched together, and the stitched result is input into the first visual encoder layer. Embed category text representations and initial text prompts. The text is concatenated, and the concatenated result is input into the first text encoder layer. ;

[0051] S33, Based on the current MapLe module Get the current text prompt word and visual cues The current visual encoder layer Output and current visual cue words The data is then stitched together, and the stitched result is input into the next visual encoder layer. ; Set the current text encoder layer Output and current text prompt words The concatenation is performed, and the concatenated result is input into the next text encoder layer. ;

[0052] MaPLe module Includes: trainable text prompts Coupled with cross-modal cue word modules; based on the MapLe module Get visual cues Includes: the currently trainable text prompts Input the cross-modal cue word coupling module to obtain visual cue words. ,Right now ;in, This represents a linear projection function.

[0053] In the initial stage, each MapLe module Trainable text prompt words The result is obtained through random initialization and subsequently iteratively updated via backpropagation based on the loss function.

[0054] The above design constructs a cross-modal bridge, allowing gradients to propagate bidirectionally between cues in both branches. This forces the semantic concepts learned by the model on the text side (such as "moss") to find corresponding fine-grained features (such as specific textures or thicknesses) on the visual side, thereby achieving stronger feature space alignment and accurately solving the problem of fine-grained recognition misalignment.

[0055] In one embodiment, the prompt length b = 2, meaning that two learnable lexical units are inserted at each level of the first J layers. The dimension of each text prompt lexical unit is... The dimensions of visual cue words are .

[0056] S34. Repeat step S33 until the J-th text encoder layer is obtained. and visual encoder layer The output;

[0057] S35, the Jth layer text encoder layer and visual encoder layer The outputs are then fed into the text encoder layer and the visual encoder layer, which are sequentially connected in series after the NJ layer, to obtain the final text encoder layer. and visual encoder layer The output, i.e., the final tongue image. visual features Text features of category j Where N is the text encoder layer. and visual encoder layer The number of floors.

[0058] The J-th layer text encoder layer and visual encoder layer The outputs are input into the NJ layer, which consists of a text encoder layer and a visual encoder layer connected in series.

[0059] S351, the Jth layer text encoder layer and visual encoder layer The outputs are respectively input into the next text encoder layer. and visual encoder layer ;

[0060] S352, Set the current text encoder layer and visual encoder layer The outputs are respectively input into the next text encoder layer. and visual encoder layer , ;

[0061] S353. Repeat step S352 until the last text encoder layer is obtained. and visual encoder layer The output.

[0062] S4. Calculate the similarity between the visual features of each tongue image and the text features of each category, and obtain the logits vector of each tongue image. Based on the logits vector of all tongue images Calculate the loss function value; where the logits vector is the log odds vector;

[0063] Calculate tongue image visual features Text features of category j Similarity includes: computational visual features Each element in the text is related to the category text features. The similarity of each element in the dataset is calculated, and the similarity of all elements is weighted and combined to obtain the final similarity score; where n is the length of the feature.

[0064] loss function ;in, Tongue image The logits vector, Tongue image The true label in the logits vector The corresponding value, Tongue image The value corresponding to the j-th category in the logits vector. Tongue image The true label, Here is a hyperparameter, and C is the number of categories.

[0065] S5. Freeze the parameters of the pre-trained CLIP model in the tongue image auxiliary diagnosis model, update the parameters of the LoRA module and the MapLe module according to the loss function value, and obtain the trained tongue image auxiliary diagnosis model when the loss function value is minimized.

[0066] The AdamW optimizer was used with a learning rate of 0.0001 and a batch size of 16. Only the LoRA, cue word, and coupling function parameters were updated, and the original weights of the pre-trained CLIP model remained frozen throughout the training process.

[0067] During inference, multimodal tongue image data is acquired, including tongue images and text prompts of all categories. The tongue images and text prompts of all categories are input into the trained tongue image-assisted diagnosis model to obtain the visual features of the tongue images and the text features of the text prompts of all categories. The similarity between the visual features of the tongue images and the text features of each category of text prompts is calculated, and the category with the highest similarity is selected as the diagnosis result.

[0068] To verify the effectiveness of this invention, comparative experiments were conducted on the TCMID-Tongue test set. Evaluation metrics included accuracy, macro-average precision, recall, and F1-score.

[0069] Comparative Experiment 1: As shown in Table 1, the present invention significantly outperforms ResNet50, DenseNet121, ViT, and MedViT models pre-trained on large-scale medical datasets in terms of accuracy. This demonstrates that the multimodal fine-tuning strategy incorporating textual semantics can effectively overcome the information bottleneck of single-modal models.

[0070] Table 1. Performance comparison of different unimodal classification networks on the TCMID-Tongue dataset.

[0071]

[0072] Comparative Experiment 2: As shown in Table 2, the present invention outperforms methods that use LoRA (CLIP-LoRA) or Adapter alone. More importantly, compared to CLIP-CoOp-LoRA and CLIP-CoCoOp-LoRA, which only perform cue learning on the text side, this method achieves deep coupling between vision and language through MapLe, improving accuracy by approximately 4%.

[0073] Table 2 Performance comparison of different efficient fine-tuning methods on the TCMID-Tongue dataset

[0074]

[0075] Ablation experiment verification: As shown in Table 3, the ablation experiment proved that LoRA and MapLe components are indispensable.

[0076] Table 3 Ablation experiments of LoRA module applied to text and visual branches

[0077]

[0078] like Figure 4 As shown, this embodiment of the invention also provides an apparatus for implementing the tongue image classification method in Embodiment 1 above, comprising:

[0079] The multimodal data acquisition module is used to acquire the tongue image to be diagnosed and the category text prompt template; the category text prompt template includes category text prompts for all categories of the tongue image;

[0080] The model loading module is used to load pre-trained tongue image auxiliary diagnostic models.

[0081] Diagnostic reasoning module: This module takes the tongue image to be diagnosed and the category text prompt template as input to the trained tongue image-assisted diagnostic model, obtains the visual features of the tongue image and the text features of each category, calculates the similarity between the visual features of the tongue image and the text features of each category, and outputs the category with the highest similarity as the final diagnostic result.

[0082] This invention also provides a terminal device, such as... Figure 5 As shown. The terminal can be a computer, medical workstation, tablet computer, or dedicated intelligent tongue diagnosis device. The terminal includes: power supply, processor, memory, input device (such as a high-definition tongue image acquisition camera), and output device (such as a display screen).

[0083] The power supply is used to power the entire terminal device.

[0084] The memory stores the computer program that implements the above-mentioned multimodal tongue image recognition-assisted diagnosis method, as well as the parameters of the trained tongue image-assisted diagnosis model (including frozen CLIP weights, trained LoRA weights, MaPLe prompt words and coupling function weights).

[0085] Input device used to acquire tongue images;

[0086] Output devices are used to display diagnostic results;

[0087] A processor is used to run computer programs stored in memory to obtain diagnostic results;

[0088] Running computer programs stored in memory includes:

[0089] The control input device acquires images of the patient's tongue and performs preprocessing (such as cropping and normalization).

[0090] Retrieve model parameters from memory;

[0091] The tongue image is input into the visual encoder, and the category text cue template is input into the text encoder to obtain the visual features of the tongue image and the text features of each category;

[0092] Calculate the similarity between the visual features of the tongue image and the text features of each category, and take the category with the highest similarity as the final diagnostic result;

[0093] The diagnosed tongue appearance category (such as "yellow and greasy coating") is displayed through the output device.

[0094] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal tongue image recognition-based auxiliary diagnostic method, characterized in that, include: Acquire multimodal tongue image data, input the multimodal tongue image data into a trained tongue image-assisted diagnostic model, and obtain diagnostic results; the training process of the tongue image-assisted diagnostic model includes: S1. Obtain the pre-trained CLIP model, construct the LoRA module and the MapLe module, and inject the LoRA module and the MapLe module into the pre-trained CLIP model to obtain the tongue image auxiliary diagnosis model; among them, the LoRA module is a low-rank matrix adaptive module, and the MapLe module is a multimodal prompt word module. S2. Obtain the multimodal tongue image dataset, which includes multiple tongue image images and category text prompt templates; the category text prompt templates include category text prompts for all categories of the tongue image images; S3. Input each tongue image and the category text prompt for each category into the tongue image auxiliary diagnosis model to obtain the visual features of each tongue image and the text features of each category; S4. Calculate the similarity between the visual features of each tongue image and the text features of each category to obtain the logits vector of each tongue image; calculate the loss function value based on the logits vector of all tongue images; where the logits vector is the log odds vector. S5. Freeze the parameters of the pre-trained CLIP model in the tongue image auxiliary diagnosis model, update the parameters of the LoRA module and the MapLe module according to the loss function value, and obtain the trained tongue image auxiliary diagnosis model when the loss function value is minimized.

2. The tongue image recognition-assisted diagnosis method based on multimodal imaging according to claim 1, characterized in that, The pre-trained CLIP model includes: a visual encoder. and text encoder Visual encoder Includes multiple layers of visual encoders connected in series. Text encoder Includes multiple layers of text encoders connected in series. Injecting LoRA and MapLe modules into the pre-trained CLIP model includes: in all visual encoder layers. and all text encoder layers Injecting a LoRA module into the self-attention module yields the visual encoder layer. and text encoder layer In each visual encoder layer of the first J layer and text encoder layer Before that, add a MapLe module. .

3. The tongue image recognition-based auxiliary diagnostic method according to claim 2, characterized in that, In the visual encoder layer Or text encoder layer Injecting LoRA modules into the self-attention module includes: Where x is the input feature of the self-attention module, , , These are the query matrix, key matrix, and value matrix injected into LoRA, respectively. These are the visual encoder layers. Or text encoder layer The projection matrix of the query matrix, key matrix, and value matrix in the self-attention module. , Let be the low-rank decomposition matrix that can be trained using the LoRA module. This is the scaling factor.

4. The multimodal tongue image recognition-based auxiliary diagnostic method according to claim 2, characterized in that, The tongue image-assisted diagnostic model processes tongue images and category text prompts in the following ways: S31. The tongue image is segmented and embedded to obtain an image embedding representation; the category text prompts are embedded with words to obtain a category text embedding representation. S32, According to the MapLe module Get the initial text prompt and initial visual cues Embedding images into representations and initial visual cues The data is stitched together, and the stitched result is input into the first visual encoder layer. Embed category text representations and initial text prompts. The text is concatenated, and the concatenated result is input into the first text encoder layer. ; S33, Based on the current MapLe module Get the current text prompt word and visual cues The current visual encoder layer Output and current visual cue words The data is then stitched together, and the stitched result is input into the next visual encoder layer. ; Set the current text encoder layer Output and current text prompt The concatenation is performed, and the concatenated result is input into the next text encoder layer. ; S34. Repeat step S33 until the J-th text encoder layer is obtained. and visual encoder layer The output; S35, the Jth layer text encoder layer and visual encoder layer The outputs are then fed into the text encoder layer and the visual encoder layer, which are sequentially connected in series after the NJ layer, to obtain the final text encoder layer. and visual encoder layer The output of; where N is the text encoder layer. and visual encoder layer The number of floors.

5. The multimodal tongue image recognition-based auxiliary diagnostic method according to claim 4, characterized in that, MaPLe module Includes: trainable text prompts Coupled with cross-modal cue word modules; based on the MapLe module Get visual cues Includes: the currently trainable text prompts Input the cross-modal cue word coupling module to obtain visual cue words. .

6. The tongue image recognition-assisted diagnosis method based on multimodal imaging according to claim 5, characterized in that, Cross-modal cue word coupling module for text cue words The processing includes: ;in, This represents a linear projection function.

7. The multimodal tongue image recognition-based auxiliary diagnostic method according to claim 1, characterized in that, Category text hint template ; where, category text hint for category j ;in, Let J be the j-th category of the tongue image, and C be the number of categories of the tongue image.

8. The multimodal tongue image recognition-based auxiliary diagnostic method according to claim 1, characterized in that, loss function ;in, Tongue image The logits vector, Tongue image The true label in the logits vector The corresponding value, Tongue image The value corresponding to the j-th category in the logits vector. Tongue image The true label, is a hyperparameter, and C is the number of categories.

9. A multimodal tongue image recognition-based auxiliary diagnostic device, used to implement the multimodal tongue image recognition-based auxiliary diagnostic method as described in any one of claims 1 to 8, characterized in that, include: The multimodal data acquisition module is used to acquire the tongue image to be diagnosed and the category text prompt template; The category text hint template includes category text hints for all categories of tongue imagery; The model loading module is used to load pre-trained tongue image auxiliary diagnostic models. Diagnostic reasoning module: This module takes the tongue image to be diagnosed and the category text prompt template as input to the trained tongue image-assisted diagnostic model, obtains the visual features of the tongue image and the text features of each category, calculates the similarity between the visual features of the tongue image and the text features of each category, and outputs the category with the highest similarity as the final diagnostic result.

10. A multimodal tongue image recognition-based auxiliary diagnostic terminal device, characterized in that, include: A power supply, used to provide power to terminal devices; A processor is used to run computer programs stored in memory to obtain diagnostic results; Input device used to acquire tongue images; Output devices are used to display diagnostic results; The memory is used to store the computer program for implementing the multimodal tongue image recognition-assisted diagnosis method as described in any one of claims 1 to 8, as well as the parameters of the trained tongue image-assisted diagnosis model.

Images