Traditional Chinese medicine tongue diagnosis medicine recommendation method based on multi-modal large model

Abstract

The present application relates to the technical field of big health and artificial intelligence, and specifically provides a traditional Chinese medicine tongue diagnosis medicine and drink recommendation method based on a multi-modal large model, which comprises the following steps: according to a pre-processed tongue image, a BLIP-2 is used as a multi-modal encoding framework to extract a complete query vector group; a visual semantic vector is constructed based on the complete query vector group; a large language model is used to generate a tongue symptom description based on the visual semantic vector; a medicine and drink knowledge base is constructed, and the tongue symptom description and user self-reports are retrieved based on the medicine and drink knowledge base to obtain final candidate medicine and drinks; and a recommendation scheme is generated based on the large language model according to the tongue symptom description and the final candidate medicine and drinks combined with a target template. The present application uses the cross-modal representation capability of the BLIP-2 model to jointly model the user text description and the tongue image features, and generates a personalized health status description based on the language model, thereby providing a more accurate semantic basis for subsequent medicine and drink recommendation.

Description

Technical Field

[0001] This invention relates to the fields of big health and artificial intelligence technologies, and more specifically, to a method for recommending herbal decoctions based on tongue diagnosis in traditional Chinese medicine using a multimodal large model. Background Technology

[0002] Tongue diagnosis in Traditional Chinese Medicine (TCM) is a crucial component of the TCM diagnostic system. It involves observing the tongue's color, shape, texture, dryness / moisture, size, positional changes, and the characteristics of its coating to help determine the functional state of the internal organs, the abundance or deficiency of Qi, blood, and body fluids, and the nature of diseases. According to TCM theory, the tongue is considered an external reflection of the heart, and the physiological functions of the five internal organs all depend on the nourishment of stomach Qi. Different dietary intakes and physical conditions can be reflected in the tongue's appearance. Therefore, the tongue's appearance can directly reveal the operational status of the internal organs. The tongue's texture primarily reflects the abundance of the organs' essence and Qi, while the tongue coating indicates the nature of pathogenic factors and the strength of stomach Qi. Thus, by conducting a detailed analysis of the tongue's appearance and combining this with the four diagnostic methods of observation, auscultation, inquiry, and palpation, doctors can gain a more comprehensive understanding of the patient's internal health, providing important evidence for clinical diagnosis and treatment. However, the clinical application of tongue diagnosis in traditional Chinese medicine in my country still faces certain limitations. Diagnostic results are influenced by various factors, including the doctor's personal clinical experience, subjective judgment, the patient's dietary habits, and the lighting and temperature of the consultation environment. Furthermore, there is a lack of unified, standardized quantitative analysis indicators and a diagnostic system. This uncertainty restricts the promotion and application of tongue diagnosis in the intelligent development of traditional Chinese medicine.

[0003] While preliminary research on intelligent tongue diagnosis in Traditional Chinese Medicine (TCM) has yielded initial results, it remains largely confined to traditional image processing and health status recognition, failing to effectively construct a complete closed-loop system encompassing "tongue image recognition—symptom inference—medicinal beverage recommendation." Significant shortcomings remain, particularly in multimodal fusion, cross-modal information alignment, and fine-grained tongue image feature analysis, hindering the implementation of personalized health intervention plans. Firstly, existing tongue image analysis methods are mostly based on shallow visual models or general computer vision architectures. Limited by lighting variations, individual differences, and image quality, their robustness in extracting fine-grained features such as color, cracks, teeth marks, and tongue coating thickness is insufficient, impacting the accuracy of subsequent health status inferences. Secondly, traditional TCM tongue diagnosis systems largely employ single-modal inference paths, failing to effectively integrate the joint semantics of user-reported symptoms and tongue image images, thus limiting the accuracy and interpretability of TCM syndrome identification. In addition, existing patents and research mainly focus on tongue image processing, diagnostic result analysis, or consultation systems, lacking practical applications of tongue diagnosis results in health intervention, especially in the personalized conditioning aspect of traditional Chinese medicine, such as herbal decoction recommendations, which is still in a blank stage. Summary of the Invention

[0004] In view of this, the present invention proposes a method for recommending Chinese medicine tongue diagnosis decoctions based on a multimodal large model, aiming to improve the diagnostic accuracy of Chinese medicine tongue diagnosis, standardize clinical diagnosis and treatment pathways, and provide feasible suggestions for promoting the development of Chinese medicine and integrated Chinese and Western medicine, so as to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above objectives, this invention proposes a method for recommending herbal decoctions based on TCM tongue diagnosis using a multimodal large model, comprising:

[0006] Based on the preprocessed tongue image, BLIP-2 was used as a multimodal coding framework to extract the complete query vector set;

[0007] Construct a visual semantic vector based on the complete query vector group;

[0008] A large-scale language model is used to generate a description of tongue symptoms based on the visual semantic vectors.

[0009] A medicinal beverage knowledge base is constructed, and based on the medicinal beverage knowledge base, the tongue appearance symptom descriptions and user self-reports are retrieved to obtain the final candidate medicinal beverages;

[0010] Based on the large-scale language model, a recommended solution is generated by combining the tongue image description, the final candidate medicinal drinks, and the target template.

[0011] Furthermore, the process of extracting tongue image features using BLIP-2 as a multimodal coding framework includes:

[0012] A feature sequence is generated from the processed tongue image based on a visual encoder;

[0013] Based on the improved Q-Former structure, information extraction and global aggregation are performed on the feature sequence to obtain a complete query vector group.

[0014] Furthermore, the improved Q-Former structure queries the visual features based on a query vector set in a multi-head cross-attention mechanism, and the query vector set is represented by the following formula:

[0015]

[0016] in, Represents a query vector group. For BridgeToken, It is the number of QueryTokens that can be learned. This represents the query vector.

[0017] Furthermore, the improved Q-Former structure includes the introduction of BridgeToken in the Q-Former network;

[0018] During the cross-attention stage, the feature sequence is weighted based on the query vector group to generate a new query vector. Key information is extracted from the feature sequence based on the new query vector. At the same time, weights are generated based on BridgeToken, and discriminative regions in the feature sequence are dynamically aggregated based on the weights.

[0019] During the self-attention phase, the new query vector is weighted and calculated to obtain the final query vector, while local detail information is absorbed through BridgeToken.

[0020] Construct a complete query vector group based on the final query vector.

[0021] Furthermore, the process of constructing a visual semantic vector based on the tongue image features includes:

[0022] The improved Q-Former structure is trained for visual-language representation based on contrastive learning and discriminative pairing methods, and visual-language generation is trained based on the language generation task of a large language model. The visual semantic vector is obtained according to the hidden state of BridgeToken in the complete query vector group.

[0023] Furthermore, the process of generating a tongue image description based on the constructed visual semantic vector includes:

[0024] The visual semantic vectors are linearly mapped to the hidden dimensions of a large language model based on the weight matrix, and a tensor is output after the mapping is constructed.

[0025] Visual-language cues are constructed based on the output tensor after the mapping.

[0026] The visual-language cues are input into the OPT model and the Flan-T5 model, respectively.

[0027] Tongue symptom descriptions are generated based on the outputs of the OPT model and the Flan-T5 model.

[0028] Furthermore, the medicinal beverage knowledge base includes the names, ingredients, preparation methods, efficacy, indications, and sources of several medicinal beverages.

[0029] Furthermore, the process of retrieving the tongue appearance description and user self-report based on the medicinal beverage knowledge base includes:

[0030] Based on the user's self-reported tongue symptoms and the vectors in the medicinal drink knowledge base, a nearest neighbor search is performed to obtain candidate medicinal drinks;

[0031] Based on the indications and efficacy fields in the medicinal beverage knowledge base, and combined with the keywords describing the tongue symptoms, the candidate medicinal beverages are subjected to Boolean filtering to determine the final candidate medicinal beverages.

[0032] Furthermore, the preprocessing of the tongue image includes:

[0033] The tongue image is subjected to boundary detection based on the target detection model, the tongue image is cropped according to the detection results, and the cropped image is standardized.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] This invention employs BLIP-2Q-Former+BridgeToken, and the model adaptively aggregates global and local visual information in cross-attention. The detection rate and F1 score of fine-grained features such as tongue coating thickness, color, cracks, and teeth marks are significantly higher than those of traditional U-Net / ViT and other simple visual networks, thus meeting the clinical demand for high-resolution diagnostic signals.

[0036] This invention directly maps the visual PromptToken to OPT or Flan-T5, generating a structured tongue image symptom description in a single forward inference. Experimental data show that the CIDEr and METEOR indices are improved by more than 10% compared to the publicly available baseline model, while the inference latency is reduced by about 30%, making it suitable for rapid deployment on mobile devices and in the cloud.

[0037] This invention transforms symptom descriptions into vector retrieval queries, combines Faiss semantic indexing to quickly locate medicinal beverage entries, and then uses fine-tuning of Flan-T5 to generate personalized treatment plans containing information such as [efficacy], [ingredients], and [source]. Each recommendation is accompanied by classical texts or medical case sources to ensure medical credibility, and can take effect immediately as the knowledge base is updated.

[0038] The method described in this invention requires only a single interaction from tongue image input to medicinal drink output, thus realizing the complete chain of "tongue image recognition → health inference → medicinal drink recommendation", which greatly improves user experience and clinical application efficiency.

[0039] This invention employs a physical separation of the vision-language backbone, knowledge base retrieval, and generation model. Upgrading any module will not affect the rest. When adding new medicines or updating medical records, only the entries need to be imported and the vector index needs to be rebuilt, without retraining the backbone network.

[0040] This invention automatically presents the compatibility principles and literature sources in the recommended text, facilitating physician review and user verification. The knowledge base can be hierarchically managed according to regulatory requirements, realizing safe and controllable intelligent TCM health services. It is superior to the existing best technologies in terms of tongue feature analysis accuracy, cross-modal generation consistency, recommendation traceability, and system maintainability, providing a more professional and reliable solution for intelligent TCM tongue diagnosis and personalized herbal intervention. Attached Figure Description

[0041] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings:

[0042] Figure 1 This is a flowchart illustrating the overall implementation of the TCM tongue diagnosis and herbal decoction recommendation method based on a multimodal large model proposed in this invention.

[0043] Figure 2 This is a schematic diagram of the improved Q-Former model structure in an embodiment of the present invention. Detailed Implementation

[0044] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0045] This embodiment proposes a TCM tongue diagnosis-based herbal decoction recommendation method based on a multimodal large model, such as... Figure 1 As shown, it includes:

[0046] Step 1: Tongue Image Acquisition and Preprocessing

[0047] To obtain clear and diagnostically valuable tongue images, this embodiment first requires users to upload a frontal photo of their tongue via a mobile device (such as a smartphone) through a mini-program. Since the original image typically contains too many irrelevant areas (such as the face and background), after receiving the image, the system calls the lightweight object detection model YOLO to accurately locate and crop the tongue region. YOLO utilizes a multi-scale feature pyramid structure and convolutional networks to detect possible tongue targets in the image in real time and outputs corresponding bounding boxes. Based on the bounding box coordinates, the system automatically crops the original image, extracting the effective tongue region as the input image for subsequent visual modeling.

[0048] The "automatic tongue localization and cropping" process implemented through YOLO not only effectively removes non-critical information such as background interference and facial occlusion, but also significantly improves the ability of subsequent visual feature extraction to focus on fine-grained features such as tongue coating thickness, color, cracks, and teeth marks. At the same time, this method considers user experience, enabling users to obtain clear and feature-rich tongue images even in non-professional shooting environments, enhancing the system's versatility and robustness.

[0049] The cropped image will undergo standardization (including normalization, size scaling, and channel arrangement adjustment) to adapt to the image input requirements of the multimodal visual coding model BLIP-2, ensuring that the model can fully capture the key structural and texture information of the tongue image during the coding stage.

[0050] Step 2: Feature Extraction from Tongue Image

[0051] This embodiment uses BLIP-2 (Bootstrapping Language-Image Pre-training v2) as the multimodal coding framework for tongue images, and introduces BridgeToken in its Q-Former to further enhance visual semantic aggregation capabilities. Its structure is as follows: Figure 2 As shown, BLIP-2 decouples the frozen visual encoder (VisionTransformer, hereinafter referred to as ViT) from the trainable query transformer (Q-Former), enabling the model to accurately capture fine-grained features of the tongue image while maintaining efficient training and seamlessly connecting with subsequent large-scale language models (OPT or Flan-T5).

[0052] Step 2.1 Visual Encoding

[0053] The normalized and resized tongue image is first input into the ViT visual encoder. ViT divides the image into fixed-size patches and maps them into high-dimensional vector sequences.

[0054]

[0055] in Used to represent the global semantics of the entire tongue image. Indicates the first Local features of an image patch It is the number of image patches output by the visual encoder. Hidden feature dimension (ViT and Q-Former are both unified at 768).

[0056] Step 2.2 Q-Former and BridgeToken Aggregation

[0057] Subsequently, the feature sequence It is fed into a trainable Q-Former. The Q-Former structure is as follows: Figure 2 As shown, Q-Former consists of a set of learnable query vectors:

[0058]

[0059] in For BridgeToken, This refers to the number of learnable QueryTokens. Under the multi-head cross-attention mechanism, Q-Former uses... right Perform information extraction and global aggregation.

[0060] (1) Cross-attention stage: from image to query

[0061] In the In layered cross-attention, the query matrix is The keys and values ​​are derived from visual features. The formulas for weight calculation and feature aggregation are as follows:

[0062]

[0063] in This is a trainable linear mapping. The bridging vector b establishes an attention connection with all patches during this process, resulting in a set of weights. Dynamically converges the discriminative regions of the entire tongue image.

[0064] (2) Self-attention stage: Information exchange between queries

[0065] Updated query Entering self-attention at the same level:

[0066]

[0067] During this phase, BridgeToken broadcasts its global semantics to the rest. Each QueryToken is processed, and its local details are absorbed to form a "global-local" fused representation.

[0068] Step 2.3 Output and LLM Interface

[0069] go through After alternating stacking of layers (cross attention + self attention + feedforward network), we obtain

[0070]

[0071]

[0072] in This is the complete query matrix that Q-Former ultimately outputs. yes The hidden state corresponding to BridgeToken serves as the visual global semantic vector. During the forward process, BridgeToken specifically learns global tongue image semantics, which can significantly enhance the capture and alignment of fine-grained features such as tongue coating thickness, color, cracks, and teeth marks. In experiments, it showed stronger cross-modal information transmission capabilities than the native BLIP-2.

[0073] Step 2.4 Cross-modal pre-training objective

[0074] BLIP-2 still uses a two-stage pre-training strategy:

[0075] Phase 1: Visual-Language Representation Learning (ITC+ITM)

[0076] • ITC (Image-Text Contrastive) contrastive learning brings paired images and text closer together, while pushing unpaired samples further away.

[0077] • ITM (Image-Text Matching) identifies matching relationships, enhancing the ability to determine the consistency between images and text.

[0078] Phase 2: Visual-to-Language Generation Learning

[0079] The output of Q-Former, after linear mapping, is used as the input of PromptEmbedding to freeze a large language model (OPT or Flan-T5), which is then further refined through a language generation task to refine the mapping from visual semantics to language space.

[0080] After completing the above pre-training, the obtained visual semantic vector, such as Equation (6) (i.e., the hidden state corresponding to BridgeToken), will be projected into the input space of LLM, providing a high-quality multimodal semantic foundation for subsequent health status description generation and RAG+T5 drug recommendation.

[0081] Step 3: Tongue Image Description Generation

[0082] After extracting features from the two tongue images, the Q-Former outputs a visual semantic vector. (Including the BridgeToken line) It needs to be seamlessly integrated with a large language model (LLM) to generate structured, medically semantically accurate descriptions of tongue symptoms. This invention supports both OPT (Decoder-Only) and Flan-T5 (Encoder-Decoder) LLMs; their access methods are as follows.

[0083] Step 3.1 Visual Prompt Construction

[0084] 1. Linear mapping

[0085]

[0086] It is the weight matrix of a single-layer linear mapping. It is the bias vector of the mapping, and its function is to... From the hidden dimension of Q-Former (=768) Hidden dimensions used when projected onto a large backend language model (OPT or Flan-T5). This achieves the matching of tensor shape and numerical range.

[0087] 2. Visual Prompt Token Group

[0088] Pick Center front The row (including the BridgeToken row) is composed of

[0089]

[0090] Should As a unified visual-linguistic prompt (VisualPrompt), it carries global tongue image semantics.

[0091] Step 3.2 Unify visual-verbal cues Input the OPT model and the Flan-T5 model respectively.

[0092] (1) OPT path (Decoder-Only)

[0093] OPT self-attention only depends on a single sequence input. Therefore, Directly concatenate to the user's text embedding front end:

[0094]

[0095] in It embeds the user-input text token, such as the first word "tongue" and the second word "image," etc.; it is directly concatenated in the OPT path. after, It is the number of visual Prompt Tokens. It is the number of user text tokens, and the multi-head self-attention of OPT in decoding the first... Step-by-step execution:

[0096]

[0097] in It includes visual prompts and historical word vectors. Since visual tokens and text tokens share the same attention matrix, the model can adaptively retrieve the global representation and local details of the tongue image when generating each word, ensuring that the symptom description is consistent with the image features.

[0098] (2) Flan-T5 path (Encoder-Decoder)

[0099] Flan-T5 processes visual and textual information separately:

[0100] 1. Visual side (Encoder Memory)

[0101] Directly output as the encoder;

[0102] 2. Text-side (Decoder Query)

[0103] User-inputted symptom text is embedded as .

[0104] In decoding the first Standard cross-attention during walking:

[0105]

[0106] in This is the current decoding hidden state. Let be the projection matrix. As a scaling factor, the global tongue image semantics carried by the BridgeToken row can be retrieved at each time step, supplementing the detailed information of the local patch query and improving the generation accuracy and interpretability.

[0107] Step 3.3 Generate Output and Examples

[0108] Through the above mechanism, LLM regressively generates a medical description of the tongue appearance, such as: "The tongue is pale red with a thin, slightly greasy white coating and obvious teeth marks, indicating a tendency of spleen deficiency and dampness, accompanied by fatigue and weakness." This output will be used as the query statement for RAG retrieval, matched with the medicinal beverage knowledge base, and finally returned as a personalized conditioning plan.

[0109] Step 4: RAG Search and Herbal Recommendations

[0110] After obtaining structured disease text through "Step 3: Tongue Description Generation," this invention outputs personalized traditional Chinese medicine decoction plans to users through Retrieval-Augmented Generation (RAG). The process consists of three steps: retrieval → filtering → generation.

[0111] Step 4.1 Knowledge Base and Index Construction

[0112] To facilitate the application of traditional Chinese medicine decoctions, this embodiment provides an Excel-formatted knowledge base of medicinal decoctions, compiling information from classic texts such as *Formulary* and *Golden Mirror of Medicine*, along with modern clinical case studies, covering more than a dozen basic medicinal decoction entries. .

[0113] Vectorization utilizes Sentence-BERT (zh), a sentence vector encoding framework based on BERT that maps complete sentences or short texts into dense vectors of fixed length, ensuring that vectors of synonymous sentences are semantically close. The "Efficacy + Applicable Certificate" field is encoded as a 768-dimensional vector and stored in the Faiss index (L2 & IVF). Faiss is an open-source vector retrieval library released by Meta AI for efficiently searching nearest neighbors in large-scale vector sets. L2 uses Euclidean distance as a similarity metric during retrieval. IVF (Inverted File) is an inverted file nearest neighbor index structure that first divides vector clusters into several "center" buckets, then performs precise distance calculations only within the closest buckets, significantly reducing search time and memory usage. The retrieval interface supports top-k similarity queries and threshold filtering (default). ).

[0114] Step 4.2 Retrieval and Candidate Item Filtering

[0115] (1) Query Construction

[0116]

[0117] Example: "Yellow and greasy tongue coating, excessive heat in the stomach and intestines; the patient reported bitter taste in the mouth and fatigue."

[0118] (2) Vector matching

[0119] Pick Perform nearest neighbor search with the knowledge base vector; return .

[0120] (3) Rule filtering

[0121] Boolean filtering is performed based on the "Indications / Efficacy" field and keywords in the LLM description (such as "clearing heat and promoting diuresis") to obtain the final candidates. .

[0122] Step 4.3 Generate personalized recommendations using Flan-T5.

[0123] The same Flan-T5 as the tongue image description (with minor adjustments made to a few "symptoms ↔ herbal decoction instructions" alignment samples) was selected as the generator.

[0124] Encoder input:

[0125] [DESCR] Yellow and greasy tongue coating... Bitter taste in the mouth;

[0126] [CAND] 1) Job's Tears and Red Bean Soup | Efficacy = Clears heat and promotes diuresis; …

[0127] 2) Sanren Tang (Three-Kernel Soup) | Efficacy = Promotes the flow of Qi;…

[0128] Decoder target template:

[0129] We recommend using {Formula Name}, {Material Compatibility Principle}, and apply to {Brief Description of Applicable Indications}.

[0130] Instructions: {Steps}. Source: {Original Source}.

[0131] The performance of the Bridge-BLIP model proposed in this invention is compared with that of the original BLIP-2 model. The results are shown in Tables 1, 2, and 3.

[0132] Table 1

[0133]

[0134] Table 2

[0135]

[0136] Table 3

[0137]

[0138] Experimental results show that the Bridge-BLIP proposed in this invention maintains or slightly outperforms the original BLIP-2 in public cross-domain retrieval and description tasks, and achieves good performance in tongue diagnosis description tasks (zero-sample CIDEr 122.1, few-sample CIDEr 143.8), verifying the effectiveness and stability of this mechanism in fine-grained medical scenarios.

[0139] Understandably, this invention proposes introducing a Bridge Token mechanism into the BLIP-2 multimodal large model structure. This mechanism, through learnable global aggregation tokens, strengthens the transmission and alignment capabilities of visual features in the language understanding process, effectively improving the model's perception accuracy and cross-modal expression capabilities for subtle pathological features in tongue images. Compared to the original BLIP-2, the introduction of Bridge Tokens achieves efficient mapping of image semantic information to the language model, effectively enhancing the aggregation and transmission capabilities of image information in the language modeling process, and significantly improving the perception and expression of fine-grained tongue features (such as teeth marks, cracks, and tongue coating). The model not only achieves cross-modal alignment and comprehensive reasoning between tongue images and user symptom descriptions, but also connects to a knowledge base of classic Chinese medicine decoctions by integrating RAG (Retrieval Enhanced Generation) technology, enabling personalized decoction recommendations based on diagnostic results, forming an intelligent closed loop of "tongue image recognition—health status analysis—treatment plan generation". Compared to previous solutions that only provided diagnostic information, this invention not only improves the accuracy and interpretability of the tongue diagnosis intelligent system, but also expands its practical application scenarios in TCM health intervention and conditioning suggestions, filling the key gap in intelligent recommendation of TCM tongue diagnosis, and providing a path with promotional value for the deep integration of traditional medicine and artificial intelligence.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for recommending herbal decoctions based on TCM tongue diagnosis using a multimodal large model, characterized in that, include: Based on the preprocessed tongue image, BLIP was used.

2. Used as a multimodal coding framework to extract complete query vector sets; Construct a visual semantic vector based on the complete query vector group; A large-scale language model is used to generate a description of tongue symptoms based on the visual semantic vectors. A medicinal beverage knowledge base is constructed, and based on the medicinal beverage knowledge base, the tongue appearance symptom descriptions and user self-reports are retrieved to obtain the final candidate medicinal beverages; Based on the large language model, a recommended solution is generated by combining the tongue image symptom description and the final candidate medicinal drinks with the target template. Using BLIP 2. The process of extracting tongue image features using a multimodal coding framework includes: A feature sequence is generated from the processed tongue image based on a visual encoder; Based on the improved Q-Former structure, information extraction and global aggregation are performed on the feature sequence to obtain a complete query vector group; The improved Q-Former structure queries visual features in a multi-head cross-attention mechanism based on a query vector set, which is represented by the following formula: ， in, Represents a query vector group. For Bridge Token It is a learnable query Token quantity Represents the query vector; The improved Q-Former structure includes the introduction of a Bridge in the Q-Former network. Token; During the cross-attention phase, the feature sequence is weighted based on the query vector group to generate a new query vector. Key information is then extracted from the feature sequence based on this new query vector, and simultaneously, based on the Bridge... Tokens generate weights, and based on these weights, discriminative regions in the feature sequence are dynamically aggregated. During the self-attention phase, the new query vector is weighted and calculated to obtain the final query vector, while simultaneously passing through the Bridge. Tokens absorb localized details; Construct a complete query vector group based on the final query vector.

2. The method for recommending herbal decoctions based on multimodal large model in traditional Chinese medicine tongue diagnosis according to claim 1, characterized in that, The process of constructing a visual semantic vector based on the tongue image features includes: The improved Q-Former structure is trained for visual-language representation based on contrastive learning and discriminative pairing methods, and further trained for visual-language generation based on a large language model language generation task, according to the Bridge in the complete query vector group. The hidden state of the token yields the visual semantic vector.

3. The method for recommending herbal decoctions based on multimodal large model in traditional Chinese medicine tongue diagnosis according to claim 1, characterized in that, The process of generating tongue image symptom descriptions based on the aforementioned construction of visual semantic vectors includes: The visual semantic vectors are linearly mapped to the hidden dimensions of a large language model based on the weight matrix, and a tensor is output after the mapping is constructed. Visual-language cues are constructed based on the output tensor after the mapping. The visual-language cues are input into the OPT model and the Flan-T5 model, respectively. Tongue symptom descriptions are generated based on the outputs of the OPT model and the Flan-T5 model.

4. The method for recommending herbal decoctions based on multimodal large model in traditional Chinese medicine tongue diagnosis according to claim 1, characterized in that, The medicinal beverage knowledge base includes the names, ingredients, preparation methods, efficacy, indications, and sources of several medicinal beverages.

5. The method for recommending herbal decoctions based on multimodal large model in traditional Chinese medicine tongue diagnosis according to claim 1, characterized in that, The process of retrieving the tongue appearance descriptions and user self-reports based on the aforementioned medicinal beverage knowledge base includes: Based on the user's self-reported tongue symptoms and the vectors in the medicinal drink knowledge base, a nearest neighbor search is performed to obtain candidate medicinal drinks; Based on the indications and efficacy fields in the medicinal beverage knowledge base, and combined with the keywords describing the tongue symptoms, the candidate medicinal beverages are subjected to Boolean filtering to determine the final candidate medicinal beverages.

6. The method for recommending herbal decoctions based on multimodal large model in traditional Chinese medicine tongue diagnosis according to claim 1, characterized in that, The preprocessing of the tongue image includes: The tongue image is subjected to boundary detection based on the target detection model, the tongue image is cropped according to the detection results, and the cropped image is standardized.

Images