Traditional Chinese Medicine Identification Methods, Systems, Electronic Devices, and Storage Media Based on Large Models

Abstract

This invention provides a method, system, electronic device, and storage medium for identifying traditional Chinese medicine (TCM) based on a large-scale model, relating to the fields of computer vision, artificial intelligence-assisted diagnosis, and drug quality control. The TCM identification technology proposed in this invention includes: first, automatically constructing a structured TCM knowledge base; then, identifying TCM materials through parallel identification and cross-validation using a master identification model and a validation model; finally, comparing the output results of the two models, and if preset conditions are met, outputting the final TCM material identification result. Compared to existing technologies, the method of this application can automatically construct a highly complete knowledge base, and on this basis, achieve high-precision, high-reliability, and professionally-oriented visual identification reports for TCM.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, artificial intelligence-assisted diagnosis, and drug quality control technology, specifically to a method, system, electronic device, and storage medium for identifying traditional Chinese medicine based on a large model. Background Technology

[0002] Accurate identification of Chinese herbal medicines is a prerequisite for ensuring safe and effective clinical use and is an indispensable core of quality control in the production, distribution and use of medicines.

[0003] In the field of traditional Chinese medicine (TCM) identification techniques, the traditional method is manual identification, which mainly relies on the professional knowledge and sensory experience of pharmaceutical experts. This is done through methods such as morphological identification (visual inspection, tactile examination, olfactory examination, and tasting), microscopic identification, and physicochemical identification. This method is time-consuming and labor-intensive, highly dependent on senior experts, and the identification results are easily affected by the expert's personal experience and state, lacking objectivity. The method of TCM image recognition based on Convolutional Neural Networks (CNN) models usually uses preprocessed image datasets collected under ideal conditions such as laboratory settings to train the model, and then uses the trained and optimized model to identify TCM. This type of method has problems such as poor model robustness, limited functionality, and strong data dependence.

[0004] In recent years, general-purpose multimodal large model (M-LLM), represented by GPT-4V and Gemini, while possessing powerful capabilities for simultaneously processing images and text, still faces the "last mile" problem when applied to the identification of traditional Chinese medicine. This is due to a lack of domain knowledge, making it difficult for general-purpose M-LLM to distinguish similar medicinal materials, assess quality grades, identify adulterants, and even prone to "illusions." Furthermore, the output of general-purpose models is usually structured and lacks traceability, failing to conform to relevant literature standards and thus unsuitable as a basis for quality control. In addition, existing automated information extraction technologies struggle to extract multimodal key information such as images and complex tables from professional literature, resulting in gaps in the knowledge base and the inability to construct a complete knowledge base for traditional Chinese medicine identification, further limiting the performance of AI.

[0005] Therefore, relying solely on a single generative large model for identification carries the risk of unreliable results and "illusions," failing to meet the stringent requirements of drug quality control. On the other hand, while a single discriminative model (such as a traditional image classifier or CLIP model) can provide classification and confidence scores, its process is opaque, unable to provide detailed identification criteria and expert-level explanations, and thus lacks credibility.

[0006] In summary, current technologies do not yet offer an automated TCM identification scheme that can organically combine the deep understanding capabilities of generative models with the quantitative verification capabilities of discriminative models to achieve internal cross-validation. Furthermore, they cannot provide a high-precision, high-reliability visual identification scheme for TCM supported by a highly complete knowledge base. Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] To address the shortcomings of existing technologies, this invention provides a method, system, electronic device, and storage medium for identifying traditional Chinese medicine based on a large model, which at least solves the problems of poor accuracy and low reliability in existing traditional Chinese medicine identification technologies.

[0009] (II) Technical Solution

[0010] To achieve the above objectives, the present invention provides the following technical solution:

[0011] Firstly, this application proposes a method for identifying traditional Chinese medicine based on a large-scale model, the method comprising:

[0012] Constructing a structured knowledge base for traditional Chinese medicine;

[0013] Asymmetric dual-model consisting of a generative master identification model and a discriminative verification model is used to perform parallel identification and cross-validation of Chinese medicinal materials. The generative master identification model is used to generate preliminary identification results based on the image of the Chinese medicinal material to be identified, and the discriminative verification model is used to obtain a list of candidate results with quantifiable confidence scores based on the image of the Chinese medicinal material to be identified and the structured Chinese medicinal knowledge base.

[0014] The preliminary identification result is compared with the candidate result corresponding to the highest confidence score in the candidate result list. If the preliminary identification result and the candidate result corresponding to the highest confidence score are consistent and the confidence score is higher than the preset verification threshold, the identification is confirmed as successful and the final identification result of Chinese medicinal materials is output.

[0015] In one embodiment, based on the image of the Chinese herbal medicine to be identified and the structured Chinese herbal medicine knowledge base, the verification model obtains a list of candidate results with quantifiable confidence scores, including:

[0016] The visual features of the Chinese herbal medicine image to be identified are encoded into an embedding vector, and compared with the text embedding vectors of all standard medicinal material names in the structured Chinese herbal medicine knowledge base, and a list of candidate results with quantifiable confidence scores is output.

[0017] In one embodiment, constructing a structured traditional Chinese medicine knowledge base includes:

[0018] Acquire and preprocess raw Chinese medicine knowledge data;

[0019] The preprocessed original Chinese medicine knowledge data is parsed using a document parsing framework.

[0020] The parsed data is processed by extracting information through two parallel information extraction pathways; the two parallel information extraction pathways include text-based structured information extraction and multimodal large model-based information extraction.

[0021] Based on a large language model, data fusion and verification are performed on the data after information extraction and processing to construct a structured Chinese medicine knowledge base.

[0022] Preferably, the document parsing framework includes, but is not limited to, the Dolphin open-source software framework.

[0023] More preferably, when using the Dolphin open-source software framework to parse the preprocessed original traditional Chinese medicine knowledge data, the process includes:

[0024] Input standardized PDF documents into the Dolphin system.

[0025] During the analysis phase, Dolphin performs comprehensive page-level layout analysis, using its built-in visual language model to identify various elements in the document and predict their natural reading order.

[0026] During the parsing phase, Dolphin converts the entire PDF document into a structured Markdown text file based on the analysis results.

[0027] In one embodiment, the text-based structured information extraction includes:

[0028] Divide the Markdown text content generated by Dolphin into blocks according to the preset character length;

[0029] For each text block, a large language model is invoked, and structured prompt words are input;

[0030] Collect and parse the JSON objects returned by the large language model to form preliminary structured data.

[0031] In one embodiment, the method further includes:

[0032] The dual model is trained using data augmentation.

[0033] In a preferred embodiment, the data augmentation includes processing the images of traditional Chinese medicine in the structured traditional Chinese medicine knowledge base as follows:

[0034] Randomly rotate, scale, and crop images of traditional Chinese medicine within a certain range, and perform geometric and color transformations; and / or

[0035] The SAM image segmentation model is used to perform high-precision foreground segmentation on images of traditional Chinese medicine; and / or

[0036] A large amount of training data on doping scenarios related to images of traditional Chinese medicine was generated using a conditional latent diffusion model.

[0037] Preferably, the conditional potential diffusion model includes, but is not limited to, the fine-tuned Stable Diffusion XL model.

[0038] In one embodiment, the method further includes:

[0039] Fine-tuning is performed on the primary identification model and the verification model; the fine-tuning includes:

[0040] A hybrid dataset is constructed to perform supervised fine-tuning of the primary identification model, and a low-rank adaptive technique is used to fine-tune the primary identification model.

[0041] A pre-trained validation model was selected as the basis for comparison and fine-tuning with the validation model.

[0042] In one embodiment, the primary discrimination model includes a multimodal large language model, which includes Qwen-2.5-VL-72B; the validation model includes a contrastive language-image pre-trained model.

[0043] In one embodiment, a structured Chinese medicine identification report is automatically generated based on the output of Chinese medicinal material identification results.

[0044] Secondly, this application also proposes a traditional Chinese medicine identification system based on a large model, the system comprising:

[0045] The knowledge base construction module is configured to build a structured traditional Chinese medicine knowledge base;

[0046] The Chinese medicine identification module is configured to identify Chinese medicinal materials based on a dual-model parallel identification and cross-validation approach; wherein, the dual models include: a main identification model for generating preliminary identification results based on the image of the Chinese medicinal material to be identified, and a validation model for obtaining a list of candidate results with quantifiable confidence scores based on the image of the Chinese medicinal material to be identified and the structured Chinese medicine knowledge base;

[0047] The identification result output module is configured to compare the preliminary identification result with the candidate result corresponding to the highest confidence score. When the preliminary identification result and the candidate result corresponding to the highest confidence score are consistent and the confidence score is higher than the preset threshold, the identification result of the Chinese medicinal material is output.

[0048] Thirdly, this application further proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the large-model-based Chinese medicine identification method as described in any of the preceding claims.

[0049] Fourthly, this application finally proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the large-model-based Chinese medicine identification method as described in any of the preceding claims.

[0050] (III) Beneficial Effects

[0051] This invention provides a method, system, electronic device, and storage medium for identifying traditional Chinese medicine based on a large-scale model. Compared with existing technologies, it has the following advantages:

[0052] 1. This application proposes a large-model-based method for identifying traditional Chinese medicine (TCM), comprising: constructing a structured TCM knowledge base; identifying TCM materials using a dual-model parallel identification and cross-validation approach; comparing the output results of the two models; and outputting the final TCM material identification result if preset conditions are met. Compared with existing technologies such as traditional manual methods, early CNN models, and general M-LLM, the method of this application can automatically construct a highly complete knowledge base and, on this basis, achieve a high-precision, high-reliability, and professionally-oriented visual identification report of TCM.

[0053] 2. The original "dual-path information extraction and fusion" method proposed in this application can simultaneously extract structured knowledge from the text and visual elements (figures and tables) of complex documents, effectively overcoming the bottlenecks of incomplete information extraction and low quality of knowledge base caused by the existing single technical path, and laying a solid data foundation for subsequent high-precision identification.

[0054] 3. The "dual-model parallel identification" architecture designed in this application introduces a cross-validation mechanism into the artificial intelligence system. By comparing the conclusions of a generative model, namely the master discriminant model (M-LLM), and a discriminant model, namely the validation model (CLIP), it can effectively identify and filter out the "illusions" and misjudgments that may occur in a single model, greatly improving the reliability and robustness of the identification results, enabling it to meet the stringent requirements of high-risk application scenarios such as drug quality control.

[0055] 4. This application significantly reduces the cost and technical threshold of system development, deployment and continuous iteration by using an automated knowledge base construction process, combining the synthesis of scarce doping scenario training data using a conditional diffusion model, and employing LoRA and other parameter fine-tuning techniques to fine-tune the dual models.

[0056] 5. The TCM identification results output by this application are no longer limited to outputting a single medicinal material name label, but can automatically generate a comprehensive, standardized, and structured identification report comparable to that of human experts. This provides front-line pharmaceutical quality control personnel with comprehensive decision support information, realizing a significant value leap from "identification tool" to "intelligent decision assistant". Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is the overall flowchart of the Chinese herbal medicine identification method based on a large model in this application;

[0059] Figure 2 This is a flowchart of a traditional Chinese medicine identification method based on a large model, as described in an embodiment of this application.

[0060] Figure 3 This is a schematic diagram of the dual-model parallel discrimination principle in the embodiments of this application;

[0061] Figure 4 This is a flowchart illustrating the core decision-making process in the embodiments of this application;

[0062] Figure 5 A bar chart showing the experimental results comparing the performance of different models;

[0063] Figure 6 This is a technical architecture diagram of the traditional Chinese medicine identification system based on a large model in the embodiments of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] This application provides a method, system, electronic device, and storage medium for identifying traditional Chinese medicine (TCM) based on a large model, which at least solves the problems of poor accuracy and low reliability in existing TCM identification technologies. The core of this method lies in combining a generative master identification model responsible for generating open-ended identification hypotheses ("what is it, why") with a discriminative verification model responsible for providing closed-set quantitative verification ("is it, how similar is it?"). Through parallel processing and mandatory cross-validation, the final output results are ensured to have both high accuracy and high reliability.

[0066] It should be noted that the large-scale model-based TCM identification technology proposed in this application is applicable to the identification of TCM and processed medicinal materials in various application scenarios. For example, it can be used by medical institutions, pharmacies, and TCM manufacturers for batch acceptance of medicinal materials; by drug testing institutes and their quality supervision departments for quality control of TCM materials; and by the general public or TCM identification enthusiasts for identification. In other words, TCM acceptance equipment and medicinal material identification apps developed based on the large-scale model-based TCM identification technology proposed in this application all fall within the technical scope of this application. To facilitate the explanation of the technical solution of this application, the following embodiments will use the example of obtaining the final identification result after a user submits a TCM identification request to illustrate the implementation process of the technical solution of this application in detail. However, this example of a specific application scenario is not intended to limit the scope of protection of this application.

[0067] Example 1:

[0068] Firstly, this invention proposes a method for identifying traditional Chinese medicine based on a large-scale model, see [link to relevant documentation]. Figure 1 The method includes:

[0069] S1. Construct a structured knowledge base for traditional Chinese medicine;

[0070] S2. Based on an asymmetric dual model consisting of a generative master identification model and a discriminative verification model, parallel identification and cross-validation of Chinese medicinal materials are performed. Among them, the generative master identification model is used to generate preliminary identification results based on the image of the Chinese medicinal material to be identified, and the discriminative verification model is used to obtain a list of candidate results with quantifiable confidence scores based on the image of the Chinese medicinal material to be identified and the structured Chinese medicinal knowledge base.

[0071] S3. Compare the preliminary identification results with the candidate results corresponding to the highest confidence score in the candidate result list. If and only if the preliminary identification results and the candidate results corresponding to the highest confidence score are consistent and the confidence score is higher than the preset verification threshold, the identification is confirmed as successful and the final identification result of the Chinese medicinal materials is output.

[0072] The following is in conjunction with the appendix Figure 1-6 The following details the implementation process of an embodiment of the present invention, including explanations of the specific steps S1-S3.

[0073] S1. Construct a structured knowledge base for traditional Chinese medicine.

[0074] To address the problems of incomplete information extraction and low knowledge base quality in existing methods of constructing traditional Chinese medicine (TCM) knowledge bases due to reliance on a single information source (plain text or plain image), which in turn leads to low accuracy and reliability in visual identification of TCM, this embodiment first proposes a dual-pathway, multimodal automated knowledge base construction method. For example... Figure 2 As shown, specifically:

[0075] S11. Acquisition and preprocessing of original Chinese medicine knowledge data.

[0076] Authoritative and professional data on traditional Chinese medicine (TCM) are acquired through web scraping or manual downloading, including but not limited to PDF versions of books related to TCM herbs. Supplementary materials such as relevant internet web pages and WeChat public account articles are also collected. To ensure data source format consistency, all non-PDF content (such as HTML web pages) is converted into standardized PDF documents using automated browser scripts (such as Selenium) that utilize the browser's "print to PDF" function, and then stored in the original document database.

[0077] S12. Based on the document parsing framework, perform document parsing on the preprocessed original Chinese medicine knowledge data.

[0078] To separate text and images from complex PDF documents, an advanced document parsing framework is used to perform in-depth preprocessing on professional PDF documents related to Chinese medicinal materials, accurately parsing them into structured Markdown text and a collection of associated independent image files.

[0079] Preferably, this embodiment uses the Dolphin open-source software framework to parse the aforementioned original traditional Chinese medicine knowledge data. Dolphin employs a novel "analyze-then-parse" two-stage processing paradigm, which is particularly suitable for complex documents with mixed text and images processed in this embodiment. The specific steps of this process are as follows:

[0080] Standardized PDF documents are input into the Dolphin system. In the first stage (analysis stage), Dolphin performs comprehensive page-level layout analysis, using its built-in visual language model to identify various elements in the document, such as text paragraphs, headings, tables, images, formulas, etc., and predict their natural reading order. In the second stage (parsing stage), based on the analysis results, Dolphin converts the entire PDF document into a structured Markdown text file. During this process, all images in the document are automatically identified, cropped, and saved as independent .jpg files. Importantly, the Markdown text retains the relative position information or citation markers of these images in the original text. Finally, a Markdown file and a series of .jpg images are produced as ideal, decoupled input data for subsequent dual-path parallel information extraction.

[0081] S13. Information extraction processing is performed on the parsed data through two parallel information extraction paths.

[0082] To maximize the extraction of all information from the original Chinese medicine knowledge data documents, two parallel and complementary information extraction pathways are used to process the parsed data. These pathways include text-based structured information extraction (Pathway A) and multimodal large-scale model-based information extraction (Pathway B). Pathway A utilizes a Large Language Model (LLM) to semantically segment and process the Markdown text content. Through carefully designed prompts, it automatically extracts key information such as the name, origin, properties, and indications of the medicines described in the text and structures it into preliminary JSON format data. Pathway B addresses non-textual information that Pathway A cannot process, utilizing a cutting-edge multimodal large-scale language model (M-LLM). Specifically:

[0083] Pathway A (Text-Based Structured Information Extraction): This path focuses on efficiently processing plain text information. First, the Markdown text content generated by Dolphin is chunked according to a preset character length (e.g., every 1000 characters) to accommodate the context window limitations of the large language model. Next, for each text chunk, a high-performance large language model (e.g., a locally deployed Qwen-2.5-VL-72B model) is called via API, with carefully designed structured prompts input. Finally, the JSON objects returned by the model are collected and parsed to form preliminary structured data.

[0084] The structured prompts include, but are not limited to, the following format and content: "You are a senior pharmaceutical expert and a data extraction robot. Your task is to extract information about traditional Chinese medicine from the following text snippet, strictly adhering to the requirements. Please extract the following fields: 'Name,' 'Source,' 'Properties,' 'Indications,' 'Dosage and Administration,' 'Precautions,' 'Identification Points,' 'Identification Methods Other than Appearance,' and 'Whether there is adulteration or counterfeiting.' Output must be in strict JSON format, where the key is the field name and the value is the corresponding content extracted from the text. If a field's content does not exist in the current text snippet, its corresponding value must be null. Do not output any explanatory text; only output the JSON object."

[0085] It should be noted that Path A can process most of the text content in a document quickly and accurately, while Path B is used to process non-text information that Path A cannot process.

[0086] Pathway B (Information Extraction Based on Multimodal Large Model): This path aims to address the inherent limitations of Pathway A, namely its inability to extract information from images, scanned tables, and charts. It is crucial for ensuring the integrity of the knowledge base. Specifically:

[0087] The original PDF document is segmented by page number (e.g., 10 pages per processing unit). For each segment, all page images (or corresponding .jpg images extracted from Dolphin) are used as image input, along with the original text or Markdown text of that segment as text input, and together they are fed into a top-level multimodal large language model.

[0088] Using cue words similar to those in pathway A, the Qwen-2.5-VL-72B model is instructed to extract information by combining its understanding of both images and text. Because the model can directly "see" and understand image content, it can extract key information that pathway A completely ignores. For example, it can extract morphological descriptions such as "cambium rings" from the annotations of a cross-sectional diagram of medicinal materials, or extract structured data from a complex comparison table of identification points.

[0089] In this embodiment, the preferred multimodal large language model is the Qwen-2.5-VL-72B model, which possesses excellent text and image understanding capabilities. In the latest technical evaluations, the Qwen-2.5-VL-72B model demonstrates industry-leading performance in document understanding, chart analysis, and structured data extraction. By directly using the original page images of the document or the extracted image files as input, the model can "read" and understand the content of tables, charts, and images, thereby extracting key identification information that cannot be obtained through a plain text approach.

[0090] S14. Based on the large language model, perform data fusion and verification on the data after information extraction and processing to construct a structured Chinese medicine knowledge base.

[0091] To integrate the results of two information extraction pathways and ensure the accuracy of the final data, this embodiment introduces an intelligent data fusion step. A large language model is used to intelligently fuse and validate the JSON data extracted from the two pathways. After comparison, merging, deduplication, and information supplementation, the large language model ultimately generates a high-quality structured traditional Chinese medicine knowledge base with information completeness far exceeding that of a single technical path. Specifically:

[0092] For the same traditional Chinese medicine (TCM) entry, two JSON datasets extracted from pathway A and pathway B are simultaneously input as context into a flagship-level large language model with powerful logical reasoning capabilities, such as GPT-4o. Through prompt word engineering, this model is instructed to act as a rigorous "data review and integration expert," with tasks including, but not limited to: 1) Merging information: merging non-repeating information from the two JSON datasets into a new JSON dataset; 2) Supplementing and refining: supplementing the macroscopic morphological description extracted from pathway A with details extracted from the image of pathway B (such as "the surface has wart-like protrusions"); 3) Deduplication and validation: identifying and removing duplicate or contradictory information, selecting more reliable and detailed descriptions; 4) Formatted output: generating a final, most complete, and validated JSON record. The final generated JSON data is stored in a structured TCM knowledge base.

[0093] It should be noted that this data fusion step ensures the completeness and accuracy of the knowledge base information, and significantly improves the quality of the automated knowledge base construction by simulating the review process of human experts.

[0094] S2. Based on an asymmetric dual model consisting of a generative master identification model and a discriminative verification model, parallel identification and cross-validation of Chinese medicinal materials are performed. The generative master identification model is used to generate preliminary identification results based on the image of the Chinese medicinal material to be identified, and the discriminative verification model is used to obtain a list of candidate results with quantifiable confidence scores based on the image of the Chinese medicinal material to be identified and the structured Chinese medicinal knowledge base.

[0095] To ensure the high reliability of the identification results, such as Figure 3 As shown in the figure, this embodiment designs a dual-model parallel identification and cross-validation algorithm. The images of Chinese medicines to be identified are fed into the main identification model and the validation model in parallel. The preliminary identification result is obtained based on the main identification model, and the candidate result list with quantifiable confidence scores is obtained based on the validation model.

[0096] During the inference phase, the images of the Chinese medicinal herbs uploaded by the user are fed in parallel into two independent models. These two independent but complementary models are the main identification model and the validation model, respectively:

[0097] The primary identification model is a multimodal large language model responsible for deep analysis of Chinese herbal medicine images, generating detailed, expert-interactive preliminary identification results. Its output includes not only the name of the medicinal material but also rich content such as morphological descriptions, key identification features, and quality assessments. In one embodiment, the primary identification model is preferably Qwen-2.5-VL-72B.

[0098] The validation model is a Contrastive Language-Image Pre-training (CLIP) model. This model is responsible for encoding the visual features of the input image into embedding vectors and performing high-precision comparisons with the text embedding vectors of all standard medicinal material names in the structured TCM knowledge base constructed above. Finally, it outputs a list of the top N candidate results with quantifiable confidence scores to quantitatively validate the text hypothesis.

[0099] It should be noted that the proposed primary identification model and verification model play asymmetric and complementary roles in this embodiment, simulating the "bold hypothesis, careful verification" process of human expert identification. The primary identification model (Qwen-2.5-VL-72B), as the "hypotheser," has the advantage of observing images like an expert and comprehensively describing its observations in natural language, including the color, texture, and cross-sectional features of the medicinal materials, and providing a comprehensive preliminary judgment based on these observations. This is an open-ended, divergent reasoning. The verification model (CLIP), as the "verifier," rigidly compares the visual features of the image with the textual concepts of all standard medicinal materials in the knowledge base, providing a convergent quantitative ranking within a closed set by calculating the cosine similarity in the vector space. The output of the primary identification model provides a target for subsequent verification, while the quantitative results of the verification model provide a solid barrier against the "illusion" risk of the primary identification model. Both are indispensable and together constitute the core of the high-reliability identification in this embodiment.

[0100] In one embodiment, to enhance the generalization ability and robustness of the dual-model approach in real-world complex application environments, all images obtained from the image knowledge base (i.e., the structured traditional Chinese medicine knowledge base) undergo a robust data augmentation pipeline before being used for training. Specifically, data augmentation includes, but is not limited to, the following processing of the images obtained from the image knowledge base:

[0101] 1) Conventional Transformations. A series of conventional geometric and color transformations are applied, including random rotation, scaling, cropping within a certain range, as well as random adjustment of the image's brightness, contrast, saturation, and hue, to simulate different shooting angles, distances, and lighting conditions.

[0102] 2) Background Replacement. This embodiment uses the Segment Anything Model (SAM) for high-precision foreground segmentation of the medicinal herb image. As a powerful basic model, SAM has excellent zero-shot segmentation capabilities, requiring no additional training for traditional Chinese medicine data to accurately separate the main subject (foreground) of the medicinal herb from its original shooting background. The segmented foreground of the medicinal herb is then randomly synthesized with a pre-built background image library containing hundreds of real-world backgrounds (such as pharmacy counters of different materials, stainless steel inspection tables, burlap sacks, wooden tabletops, white background boards, etc.).

[0103] It should be noted that this step is crucial for dealing with background interference. This step forces the model to focus on learning the inherent and stable identification features of the medicinal material itself (such as texture, color, and shape), while ignoring variable and irrelevant background information, thereby significantly improving its performance in practical applications.

[0104] 3) Generation of synthetic data for adulterated scenarios. Collecting large quantities of adulterated samples of traditional Chinese medicine in a systematic manner in the real world is a major bottleneck for model training. This embodiment employs a Conditional Latent Diffusion Model to programmatically generate massive amounts of photorealistic training data for adulterated scenarios. Preferably, the Conditional Latent Diffusion Model includes, but is not limited to, a Stable DiffusionXL model fine-tuned on a refined dataset of medicinal herb images.

[0105] The content of the generated images can be precisely controlled through structured prompt engineering. For example, the following prompt can be input into the model: "A high-resolution photograph of Chinese medicinal herbs, mainly consisting of 85% genuine Astragalus membranaceus slices, randomly mixed with 15% of similar-looking counterfeit Caragana root slices. All slices are randomly scattered in a white ceramic dish, photographed using standard laboratory top lighting and a macro lens, with clear details and photorealistic quality."

[0106] For each synthesized image generated, the system will synchronously create a corresponding structured label, for example: {"components": [{"name": "Astragalus membranaceus", "ratio": 0.85}, {"name": "Caragana korshinskii root", "ratio":0.15}], "is_adulterated": true}.

[0107] Adding these massive amounts of "synthetic image-label" data pairs to the training dataset can effectively teach the master discrimination model (Qwen-2.5-VL-72B) not only to identify the presence of multiple components, but also to accurately describe the doped substances and their proportions.

[0108] In one embodiment, in order to reduce the reliance on large-scale computing resources (such as multi-GPU clusters) and improve model iteration efficiency, a parameter-efficient fine-tuning technique is used to fine-tune the aforementioned master identification model and verification model.

[0109] Fine-tuning of the master discrimination model (Qwen-2.5-VL-72B):

[0110] Supervised fine-tuning (SFT) was performed using a carefully constructed hybrid dataset that included: text-image pairs constructed from a knowledge base, the aforementioned synthetic mixed-scene data, and dialogue-formatted data simulating user interactions.

[0111] Low-Rank Adaptation (LoRA) was employed to fine-tune the master discriminator model (Qwen-2.5-VL-72B). The core idea of ​​LoRA is to freeze the vast majority (over 99%) of the weights in the pre-trained model, injecting only small, trainable low-rank matrices into key parts of the model (such as the attention module in the Transformer layer). During training, only the parameters of these newly added low-rank matrices are updated. This method can reduce the GPU memory requirements for training by several times, making it possible to perform domain-knowledge-based fine-tuning of ultra-large 72B models on a single or small number of consumer / professional GPUs.

[0112] Validation Model (CLIP) Fine-tuning:

[0113] Choose a high-performance pre-trained CLIP model as a base, such as ViT-L / 14 from the OpenCLIP project.

[0114] Fine-tuning is performed using contrastive learning on data pairs consisting of (images of medicinal materials, text of standard medicinal material names). The objective function of training is to maximize the cosine similarity of matching image-text pairs in the embedding space, while minimizing the similarity of mismatched image-text pairs.

[0115] After fine-tuning, the CLIP model became a highly specialized "Chinese medicine validator". Its embedding space can accurately reflect the semantic relationship between the visual characteristics of Chinese medicinal materials and their standard names, providing a reliable quantitative basis for subsequent cross-validation.

[0116] S3. Compare the preliminary identification results with the candidate results corresponding to the highest confidence score in the candidate result list. If and only if the preliminary identification results and the candidate results corresponding to the highest confidence score are consistent and the confidence score is higher than the preset verification threshold, the identification is confirmed as successful and the final identification result of the Chinese medicinal materials is output.

[0117] The system compares and arbitrates the outputs of the two models. Only when the judgment of the primary identification model matches the highest confidence result of the verification model, and the confidence score is higher than a preset threshold, will the system output the final identification conclusion. If the two results are inconsistent or the confidence score is low, a "fuzziness alert" is issued, prompting the user to conduct further manual verification, thereby greatly improving the security and reliability of the system in practical applications.

[0118] In one embodiment, a structured TCM identification report is automatically generated to directly and intuitively present the final identification conclusion of TCM.

[0119] Based on the highly reliable results verified by the dual-model identification algorithm, the system automatically retrieves all information related to the medicinal material from the structured knowledge base and integrates the real-time analysis conclusions of the master identification model. Finally, the system automatically generates a structured auxiliary identification report for traditional Chinese medicine quality. This report is comprehensive, covering the name, source, properties, indications, key identification points, possible adulteration situations and their identification methods, and can indicate the source of information. This function realizes a paradigm shift from providing "simple classification labels" to issuing "professional consultation opinions," providing users with direct, usable, and professional decision support.

[0120] This completes the entire process of the large-scale model-based method for identifying traditional Chinese medicine proposed in this embodiment and its preferred embodiments.

[0121] The following example uses a real-world application scenario where a user uploads an image of the Chinese herb to be identified (Angelica sinensis image) and requests identification, and then obtains the identification result. This example will be used to explain the specific implementation process of the Chinese herb identification method based on a large model proposed in this embodiment and its preferred embodiments.

[0122] 1) Dual-model parallel inference.

[0123] When a user uploads an image of a Chinese herbal medicine to be identified via the client, the image data is sent to the backend inference engine and distributed to two parallel processing pathways:

[0124] Pathway 1 (Primary Identification Model Inference): The image of the Chinese herbal medicine to be identified is input into the finely tuned Qwen-2.5-VL-72B model. The primary identification model performs a forward propagation, generating a detailed natural language description. For example: "Based on image analysis, this sample is preliminarily identified as 'Angelica sinensis' slices. Its surface is yellowish-brown, the cross-section is yellowish-white, and there are brown oil spots, which matches the description of Angelica sinensis in a certain professional literature. The sample slices are of uniform thickness, with no obvious signs of mold, indicating good quality." The system then uses rule or intent recognition to parse the preferred herbal name "Angelica sinensis" from this text.

[0125] Pathway 2 (Verification Model Inference): The image of the Chinese herbal medicine to be identified is input into a finely tuned CLIP model, which encodes it into a high-dimensional image embedding vector. This vector is fed into a vector database, where batch cosine similarity calculations are performed against the text embedding vectors of hundreds of standard medicinal material names pre-calculated and stored in a structured Chinese herbal medicine knowledge base. The database returns a candidate list sorted from high to low similarity scores, for example: [("Angelica sinensis", 0.95), ("Angelica dahurica", 0.78), ("Angelica dahurica", 0.65),...].

[0126] (ii) Confidence assessment and core decision-making.

[0127] To obtain more reliable and interpretable confidence scores and implement rigorous decision-making logic, perform the following steps:

[0128] Confidence calibration:

[0129] Directly using cosine similarity scores in the range [-1, 1] as probabilities is inaccurate. In this embodiment, a temperature-scaled softmax function is used to calibrate the cosine similarity score (sim) output by the CLIP model, converting it into a more statistically meaningful confidence probability (P). The calculation formula is as follows:

[0130] P(i) = \frac{\exp(\text{sim}(i) / T)}{\sum_{j} \exp(\text{sim}(j) / T)}

[0131] It can also be expressed as:

[0132]

[0133] in:

[0134] Represents the confidence probability of the i-th candidate medicinal material. P(i) is a value between 0 and 1, and the sum of P values of all candidate medicinal materials is 1; i represents the index of the candidate medicinal material. For example, in the candidate list [("Angelica sinensis", 0.95), ("Levisticum officinale", 0.78), ("Angelica dahurica", 0.65),...], the index i of "Angelica sinensis" can be 1, the index i of "Levisticum officinale" can be 2, and so on.

[0135] sim(i) represents the cosine similarity score between the image to be identified and the i-th candidate medicinal material calculated by the CLIP model (verification model). The higher this value, the more similar the model thinks the image and the medicinal material name are in terms of features, and its original range is between [-1, 1].

[0136] T represents a hyperparameter called temperature, which is a key parameter for adjusting the smoothness of the probability distribution. When the value of T is low, it will make the probability distribution more "sharp", that is, magnify the gap between the high-score and the second-high-score options, making the model more confident in the highest-score option. When T the value is high, it will make the probability distribution smoother, that is, narrow the probability gap between each option. T The value of is determined by experimental tuning on an independent validation set to make the output probability distribution best reflect the true credibility.

[0137] j represents the index variable for traversing all candidate results. According to the standard definition of the softmax function, the role of j here is to traverse each term in the sum in the denominator. That is, if there are N medicinal materials in the candidate list, j will traverse from 1 to N to ensure that the denominator is the normalization factor of the sum of scores of all candidates.

[0138] , the sum, means adding up all the results after calculating exp(sim(j) / T) for all candidate medicinal materials (traversed by index j). This sum is the normalization constant, which is used to ensure that the sum of all calculated P(i) is equal to 1, thus forming a valid probability distribution.

[0139] Core decision:

[0140] The decision-making process is as follows Figure 4 shown, including consistency check and similarity risk assessment.

[0141] Consistency check: Compare the preferred name resolved by the master identification model (pathway 1 result) with the Top-1 candidate name after calibration by the validation model (pathway 2 result). If they are inconsistent, it is immediately judged as "result ambiguous". At this time, an alarm message will be generated, which reads: "The system has detected a conflict between the results of two independent analysis models and cannot give a highly reliable identification conclusion. It is recommended that you submit this sample to a pharmacology expert for manual review." If they are consistent, proceed to the next step of confidence check.

[0142] Similarity Risk Assessment: Check whether the confidence probability of the Top-2 candidates in the validation model exceeds a preset risk threshold (e.g., P(Top-2) > 0.2). If it exceeds this threshold, it indicates that the sample is highly similar in appearance to another medicinal material, posing a risk of confusion. The system will attach a clear warning message to the final report: "Note: This sample is similar to '..." "If the appearances are highly similar, please carefully distinguish them in conjunction with other identification points (such as smell, physicochemical identification) to avoid confusion." If the confidence probability of the Top-2 candidates does not exceed this threshold, it is considered a successful identification with high confidence.

[0143] (iii) Automated report generation

[0144] After passing all the above verifications and being determined as a highly reliable identification, the automated report generation process is initiated:

[0145] Using the identified medicinal material name (such as "Angelica sinensis") as a unique identifier, a precise search is performed in a structured knowledge base to obtain all related structured information (name, source, properties, indications, dosage, precautions, identification points, adulteration, etc.).

[0146] The authoritative, static information retrieved from the knowledge base is intelligently integrated with the real-time, dynamic analysis conclusions generated by the main identification model (such as "the sample surface has xxx characteristics, which meet the superior product standard") and the quantitative confidence data given by the verification model.

[0147] Finally, the system calls a report generation engine (such as a template-based PDF rendering library) to populate the integrated information into a preset professional report template, generating a well-formatted PDF document containing both text and images for auxiliary identification of traditional Chinese medicine quality. This document is then returned to the client for users to view, download, and archive. The report clearly indicates the source of key information (such as morphological descriptions), greatly enhancing its authority and practical value.

[0148] To objectively and quantitatively evaluate the effectiveness and superiority of the "Dual-Model-Based Method for Identifying Traditional Chinese Medicine" (hereinafter referred to as "this invention") proposed in this embodiment, a set of comparative experiments was designed. The aim was to compare the method of this embodiment with existing mainstream technologies (such as CNN-based classifiers and general multimodal large models) on multiple industry standard evaluation metrics, particularly their performance when handling easily confused or mixed complex samples. See Table 1 and Appendix below. Figure 5 The method proposed in this embodiment is compared with existing mainstream technologies.

[0149] Table 1: Comparison and Analysis of the Invention and the Prior Art

[0150]

[0151] Based on Table 1 and Appendix above Figure 5 It is evident that the technology presented in this application surpasses existing related technologies in all aspects, including knowledge base construction, specific identification, identification result output, and result reliability. Overall, the method achieves higher accuracy and reliability in the identification of traditional Chinese medicine compared to existing technologies.

[0152] Example 2:

[0153] Secondly, this invention also provides a traditional Chinese medicine identification system based on a large model, see [link to relevant documentation]. Figure 6 The system mainly includes:

[0154] The knowledge base construction module is configured to build a structured traditional Chinese medicine knowledge base;

[0155] The Chinese medicine identification module is configured to identify Chinese medicinal materials based on a dual-model parallel identification and cross-validation approach; wherein, the dual models include: a main identification model for generating preliminary identification results based on the image of the Chinese medicinal material to be identified, and a validation model for obtaining a list of candidate results with quantifiable confidence scores based on the image of the Chinese medicinal material to be identified and the structured Chinese medicine knowledge base;

[0156] The identification result output module is configured to compare the preliminary identification result with the candidate result corresponding to the highest confidence score. When the preliminary identification result and the candidate result corresponding to the highest confidence score are consistent and the confidence score is higher than the preset threshold, the identification result of the Chinese medicinal material is output.

[0157] In practical implementation, this embodiment proposes a traditional Chinese medicine identification system based on a large model, whose system architecture includes three parts: a client, a backend server, and a database cluster. Specifically:

[0158] Client: An application that can be deployed on desktop operating systems (such as Windows, macOS) or mobile devices (such as iOS, Android). This client provides users with a user interface for image capture / uploading, text query input, identification report visualization, and interactive dialogue with the backend model.

[0159] Backend Server: A server cluster deployed in the cloud (such as Alibaba Cloud, AWS) or a local data center, hosting the core computing and logic of the system. During runtime, it executes the steps of the large-model-based traditional Chinese medicine identification method described in Embodiment 1 and its preferred embodiments, and mainly includes the following functional modules:

[0160] The knowledge base construction module is configured to build a structured Chinese medicine knowledge base based on the acquired and preprocessed raw Chinese medicine knowledge data.

[0161] The model training and augmentation module is configured to augment the model training data and to optimize the training of the dual models using the augmented data.

[0162] The Inference Engine Module, also known as the Chinese Medicine Identification Module, is configured to identify Chinese medicinal materials using a dual-model parallel identification and cross-validation approach.

[0163] The report generation module, also known as the identification result output module, is used to integrate information and generate a PDF report.

[0164] Database Cluster:

[0165] Raw Document Repository: Uses object storage services (such as OSS) to store unstructured raw data such as collected PDFs and web archives;

[0166] Image Knowledge Base: Uses a distributed file system to store high-quality image files extracted from literature and associated with each Chinese medicine entry.

[0167] Structured Knowledge Base: Uses a document database (such as MongoDB) to store JSON-formatted information on traditional Chinese medicine that has been extracted, fused, and validated using a dual-pathway (pathway A and path B) method.

[0168] Vector Database: This uses a dedicated vector database (such as Milvus or FAISS) to efficiently store and retrieve the text embedding vectors of medicinal herb names for CLIP model validation.

[0169] It is understood that the large-model-based Chinese medicine identification system provided in this embodiment of the invention corresponds to the large-model-based Chinese medicine identification method described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the large-model-based Chinese medicine identification method, and will not be repeated here.

[0170] Example 3:

[0171] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the large-model-based method for identifying traditional Chinese medicine as described in any of the above embodiments and their preferred embodiments. The method mainly includes:

[0172] S1. Construct a structured knowledge base for traditional Chinese medicine;

[0173] S2. Identify Chinese medicinal materials based on a dual-model parallel identification and cross-validation method, wherein the dual models include: a master identification model for generating preliminary identification results and a validation model for outputting a list of candidate results with quantifiable confidence scores;

[0174] S3. Compare the preliminary identification results with the candidate results corresponding to the highest confidence score. When the preliminary identification results and the candidate results corresponding to the highest confidence score are consistent and the confidence score is higher than the preset threshold, output the identification results of Chinese medicinal materials.

[0175] It is understood that the electronic device for identifying traditional Chinese medicine based on a large model provided in this embodiment of the invention corresponds to the above-mentioned method and system for identifying traditional Chinese medicine based on a large model. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the method and system for identifying traditional Chinese medicine based on a large model, and will not be repeated here.

[0176] Example 4:

[0177] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the large-model-based method for identifying traditional Chinese medicine as described in any of the above embodiments and their preferred embodiments, the method comprising:

[0178] S1. Construct a structured knowledge base for traditional Chinese medicine;

[0179] S2. Identify Chinese medicinal materials based on a dual-model parallel identification and cross-validation method, wherein the dual models include: a master identification model for generating preliminary identification results and a validation model for outputting a list of candidate results with quantifiable confidence scores;

[0180] S3. Compare the preliminary identification results with the candidate results corresponding to the highest confidence score. When the preliminary identification results and the candidate results corresponding to the highest confidence score are consistent and the confidence score is higher than the preset threshold, output the identification results of Chinese medicinal materials.

[0181] It is understood that the large-model-based Chinese medicine identification storage medium provided in this embodiment of the invention corresponds to the large-model-based Chinese medicine identification method and system described above. The explanations, examples, and beneficial effects of its contents can be referred to the corresponding contents in the large-model-based Chinese medicine identification method and system, and will not be repeated here.

Claims

1. A large model-based traditional Chinese medicine identification method, characterized in that, The method includes: Constructing a structured knowledge base for traditional Chinese medicine; Asymmetric dual-model consisting of a generative master identification model and a discriminative verification model is used to perform parallel identification and cross-validation of Chinese medicinal materials. The generative master identification model is used to generate preliminary identification results based on the image of the Chinese medicinal material to be identified, and the discriminative verification model is used to obtain a list of candidate results with quantifiable confidence scores based on the image of the Chinese medicinal material to be identified and the structured Chinese medicinal knowledge base. The identification process involves comparing the preliminary identification result with the candidate result corresponding to the highest confidence score in the candidate result list. If the preliminary identification result and the candidate result corresponding to the highest confidence score are consistent and the confidence score is higher than a preset verification threshold, the identification is confirmed as successful and the final identification result of the Chinese medicinal material is output. This process includes: if the preliminary identification result and the candidate result corresponding to the highest confidence score in the candidate result list are inconsistent, a manual review request is generated; if they are consistent, a confidence check is performed: comparing whether the second highest confidence score exceeds the verification threshold; if it exceeds the verification threshold, a risk warning of high similarity of medicinal materials is generated; if it does not exceed the verification threshold, the identification is confirmed as successful and the final identification result of the Chinese medicinal material is output. The construction of a structured traditional Chinese medicine knowledge base includes: Acquire and preprocess raw Chinese medicine knowledge data; The preprocessed original Chinese medicine knowledge data is parsed using a document parsing framework. The parsed data is processed by extracting information through two parallel information extraction pathways; the two parallel information extraction pathways include text-based structured information extraction and multimodal large model-based information extraction. Based on a large language model, data fusion and verification are performed on the data after information extraction and processing to construct a structured Chinese medicine knowledge base.

2. The method of claim 1, wherein, Based on the image of the Chinese herbal medicine to be identified and the structured Chinese herbal medicine knowledge base, the validation model obtains a list of candidate results with quantifiable confidence scores, including: The visual features of the Chinese herbal medicine image to be identified are encoded into an embedding vector, and compared with the text embedding vectors of all standard medicinal material names in the structured Chinese herbal medicine knowledge base, and a list of candidate results with quantifiable confidence scores is output.

3. The method of claim 1, wherein, The method further includes: The dual model is trained using data augmentation.

4. The method of claim 3, wherein, The data augmentation includes processing the images of Chinese herbs in the structured Chinese herbal medicine knowledge base as follows: Randomly rotate, scale, and crop images of traditional Chinese medicine within a certain range, and perform geometric and color transformations; and / or The SAM image segmentation model is used to perform high-precision foreground segmentation on images of traditional Chinese medicine; and / or A large amount of training data on doping scenarios related to images of traditional Chinese medicine was generated using a conditional latent diffusion model.

5. The method of claim 1, wherein, The method further includes: Fine-tuning is performed on the primary identification model and the verification model; the fine-tuning includes: A hybrid dataset is constructed to perform supervised fine-tuning of the primary identification model, and a low-rank adaptive technique is used to fine-tune the primary identification model. A pre-trained validation model was selected as the basis for comparison and fine-tuning with the validation model.

6. The method of claim 1, wherein, The primary discrimination model includes a multimodal large language model, which includes Qwen-2.5-VL-72B; the validation model includes a contrastive language-image pre-trained model.

7. A large model-based traditional Chinese medicine identification system, characterized in that, The system includes: The knowledge base construction module is configured to build a structured traditional Chinese medicine knowledge base; The Chinese medicine identification module is configured to identify Chinese medicinal materials based on a dual-model parallel identification and cross-validation approach; wherein, the dual models include: a main identification model for generating preliminary identification results based on the image of the Chinese medicinal material to be identified, and a validation model for obtaining a list of candidate results with quantifiable confidence scores based on the image of the Chinese medicinal material to be identified and the structured Chinese medicine knowledge base; The identification result output module is configured to compare the preliminary identification result with the candidate result corresponding to the highest confidence score. When the preliminary identification result and the candidate result corresponding to the highest confidence score are consistent and the confidence score is higher than a preset threshold, the identification result of the Chinese medicinal material is output. This process includes: if the preliminary identification result and the candidate result corresponding to the highest confidence score in the candidate result list are inconsistent, a manual review request is generated; if they are consistent, a confidence check is performed: the second highest confidence score is compared to see if it exceeds the verification threshold; if it exceeds the verification threshold, a risk warning of high similarity of medicinal materials is generated; if it does not exceed the verification threshold, the identification is confirmed as successful and the final identification result of the Chinese medicinal material is output. The construction of a structured traditional Chinese medicine knowledge base includes: Acquire and preprocess raw Chinese medicine knowledge data; The preprocessed original Chinese medicine knowledge data is parsed using a document parsing framework. The parsed data is processed by extracting information through two parallel information extraction pathways; the two parallel information extraction pathways include text-based structured information extraction and multimodal large model-based information extraction. Based on a large language model, data fusion and verification are performed on the data after information extraction and processing to construct a structured Chinese medicine knowledge base.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the traditional Chinese medicine identification method based on a large model as described in any one of claims 1-6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the large-model-based method for identifying traditional Chinese medicine as described in any one of claims 1-6.

Images