Traditional Chinese medicine identification method and system based on large model, electronic equipment and storage medium

Abstract

The invention provides a traditional Chinese medicine identification method and system based on a large model, electronic equipment and a storage medium, and relates to the technical field of computer vision, artificial intelligence auxiliary diagnosis and medicine quality control. The invention provides a traditional Chinese medicine identification technology based on a large model, which comprises the following steps: firstly, automatically constructing a structured traditional Chinese medicine knowledge base, and then identifying traditional Chinese medicines based on a double-model parallel identification and cross validation mode of a main identification model and a verification model; and finally, comparing output results of the double models, and if a preset condition is met, outputting a final traditional Chinese medicinal material identification result. Compared with the prior art, the method has the advantages that the high-integrity knowledge base can be automatically constructed, and high-precision, high-reliability and content-specialized traditional Chinese medicine visual identification reports can be output on the basis of the high-integrity knowledge base.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, artificial intelligence assisted diagnosis and drug quality control, and in particular to a traditional Chinese medicine identification method and system based on a large model, an electronic device and a storage medium. BACKGROUND

[0002] Accurate identification of traditional Chinese medicine is the premise of ensuring the safety and effectiveness of clinical medication, and is an indispensable quality control core in the production, circulation and use of drugs.

[0003] In the field of traditional Chinese medicine identification technology, the traditional method is manual identification, which mainly relies on the professional knowledge and sensory experience of pharmaceutical experts, and is carried out through means such as appearance identification (visual observation, hand touch, nose smell, mouth taste), microscopic identification and physicochemical identification. This method is time-consuming and labor-intensive, highly dependent on experienced experts, and the identification results are easily affected by the personal experience and state of the experts, lacking objectivity; the method of identifying traditional Chinese medicine based on a convolutional neural network (CNN) model usually uses an image dataset collected under ideal laboratory conditions and preprocessed to train the model, and then uses the trained and optimized model to identify traditional Chinese medicine. This type of method has problems such as poor model robustness, single function, and strong data dependency.

[0004] In recent years, general multi-modal large language models (M-LLM) represented by GPT-4V and Gemini have powerful capabilities to process images and text simultaneously, but when applied to traditional Chinese medicine identification, the "last mile" problem is still prominent. This is because of the lack of domain knowledge, making it difficult for general M-LLM to distinguish between similar medicinal materials, evaluate quality grades, identify adulterants, and also prone to "hallucinations"; at the same time, the output of general models is usually structured and has no traceability, which does not meet the relevant literature specifications and is difficult to serve as a quality control basis. In addition, existing automatic information extraction technology is difficult to extract multi-modal key information such as pictures and complex tables in professional literature, resulting in missing knowledge in the knowledge base, and the problem of being unable to build a complete traditional Chinese medicine identification knowledge base, further limiting the performance of AI.

[0005] Therefore, relying solely on a single generative large model for identification poses the risk of unreliable results and "hallucinations", and cannot meet the stringent requirements of drug quality control; while a single discriminative model (such as a traditional image classifier or CLIP model) can give classification and confidence, but its process is not transparent, cannot provide detailed identification basis and expert-level explanation, and lacks credibility.

[0006] In summary, there is no existing technology that can combine the deep understanding of generative models with the quantitative verification ability of discriminative models, realize internal cross-validation, and provide a high-precision and high-reliability visual identification scheme for traditional Chinese medicine under the support of a high-integrity knowledge base. SUMMARY

[0007] (I) Technical problems to be solved In view of the deficiencies of the prior art, the present application provides a traditional Chinese medicine identification method and system based on a large model, an electronic device and a storage medium, which at least solves the problems of poor precision and low reliability of existing traditional Chinese medicine identification technology.

[0008] (II) Technical solutions To achieve the above object, the present application is implemented by the following technical solutions: In a first aspect, the present application first proposes a traditional Chinese medicine identification method based on a large model, which comprises: Constructing a structured traditional Chinese medicine knowledge base; Using an asymmetric double model composed of a generative main identification model and a discriminative verification model to perform parallel identification and cross-validation on traditional Chinese medicinal materials; wherein the generative main identification model is used to generate a preliminary identification result based on a to-be-identified traditional Chinese medicine image, and the discriminative verification model is used to obtain a candidate result list with quantifiable confidence scores based on the to-be-identified traditional Chinese medicine image and the structured traditional Chinese medicine knowledge base; Comparing the preliminary identification result with the candidate result corresponding to the highest confidence score in the candidate result list, and only when the preliminary identification result and the candidate result corresponding to the highest confidence score are consistent and the confidence score is higher than a preset verification threshold, the identification is confirmed to be successful and the final traditional Chinese medicine material identification result is output.

[0009] In one embodiment, based on the to-be-identified traditional Chinese medicine image and the structured traditional Chinese medicine knowledge base, the verification model obtains a candidate result list with quantifiable confidence scores, which comprises: Encode the visual features of the to-be-identified traditional Chinese medicine image into an embedding vector, and compare it with the text embedding vectors of all standard medicinal material names in the structured traditional Chinese medicine knowledge base, to output a candidate result list with quantifiable confidence scores.

[0010] In one embodiment, constructing a structured traditional Chinese medicine knowledge base comprises: Obtain raw traditional Chinese medicine knowledge data and preprocess it; Perform document parsing on the preprocessed raw traditional Chinese medicine knowledge data based on a document parsing framework; Perform information extraction processing on the parsed data through two parallel information extraction channels; the two parallel information extraction channels comprise text-based structured information extraction and multi-modal large model-based information extraction; Based on a large language model, the data after information extraction processing is fused and verified to construct a structured traditional Chinese medicine knowledge base.

[0011] Preferably, the document parsing framework includes but is not limited to the Dolphin open source software framework.

[0012] More preferably, when the pre-processed original traditional Chinese medicine knowledge data is parsed by the Dolphin open source software framework, it includes: The standardized PDF document is input into the Dolphin system, In the analysis stage, Dolphin performs comprehensive page-level layout analysis, identifies various elements in the document using its built-in visual language model, and predicts their natural reading order; In the parsing stage, Dolphin converts the entire PDF document into a structured Markdown format text file based on the analysis results.

[0013] In one embodiment, the text-based structured information extraction includes: The Dolphin-generated Markdown text content is divided into blocks according to the preset character length; For each text block, a large language model is called and a structured prompt word is input; Collect and parse the JSON object returned by the large language model to form preliminary structured data.

[0014] In one embodiment, the method further includes: The dual model is trained using data enhanced data.

[0015] In a preferred embodiment, the data enhancement includes the following processing on the traditional Chinese medicine pictures in the structured traditional Chinese medicine knowledge base: Randomly rotate, scale, crop, and perform geometric and color transformations on the traditional Chinese medicine pictures within a certain range; and / or Use the SAM image segmentation model to perform high-precision foreground segmentation on the traditional Chinese medicine pictures; and / or Use the conditional latent diffusion model to generate a large number of traditional Chinese medicine picture-related adulteration scene training data.

[0016] Preferably, the conditional latent diffusion model includes but is not limited to the fine-tuned Stable Diffusion XL model.

[0017] In one embodiment, the method further includes: Fine-tuning the main discrimination model and the verification model; the fine-tuning includes: constructing a mixed dataset to supervise fine-tuning of the main discrimination model and fine-tuning the main discrimination model using a low-rank adaptive technology; selecting a pre-training verification model as a basis to compare learning fine-tuning with the verification model.

[0018] In one embodiment, the main discrimination model comprises a multi-modal large language model, and the multi-modal large language model comprises Qwen-2.5-VL-72B; and the verification model comprises a contrastive language-image pre-training model.

[0019] In one embodiment, a structured Chinese medicine identification report is automatically generated based on the output Chinese medicinal material discrimination result.

[0020] In a second aspect, the present application further provides a Chinese medicine discrimination system based on a large model, comprising: a knowledge base construction module configured to construct a structured Chinese medicine knowledge base; a Chinese medicine discrimination module configured to discriminate Chinese medicinal materials based on a double model parallel discrimination and cross-validation manner; wherein the double model comprises: a main discrimination model for generating a preliminary identification result based on a to-be-discriminated Chinese medicine image, and a verification model for obtaining a candidate result list with quantifiable confidence scores based on the to-be-discriminated Chinese medicine image and the structured Chinese medicine knowledge base; a discrimination result output module configured to compare the preliminary identification result and the candidate result corresponding to the highest confidence score, and output a Chinese medicinal material discrimination result when the preliminary identification result and the candidate result corresponding to the highest confidence score are consistent and the confidence score is higher than a preset threshold.

[0021] In a third aspect, the present application further provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the Chinese medicine discrimination method based on a large model according to any one of the above embodiments.

[0022] In a fourth aspect, the present application finally provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the Chinese medicine discrimination method based on a large model according to any one of the above embodiments.

[0023] (Three) beneficial effects The present application provides a Chinese medicine discrimination method, system, electronic device and storage medium based on a large model. Compared with the prior art, the following beneficial effects are achieved: 1. This application proposes a large-model-based method for identifying traditional Chinese medicine (TCM), comprising: constructing a structured TCM knowledge base; identifying TCM materials using a dual-model parallel identification and cross-validation approach; comparing the output results of the two models; and outputting the final TCM material identification result if preset conditions are met. Compared with existing technologies such as traditional manual methods, early CNN models, and general M-LLM, the method of this application can automatically construct a highly complete knowledge base and, on this basis, achieve a high-precision, high-reliability, and professionally-oriented visual identification report of TCM.

[0024] 2. The original "dual-path information extraction and fusion" method proposed in this application can simultaneously extract structured knowledge from the text and visual elements (figures and tables) of complex documents, effectively overcoming the bottlenecks of incomplete information extraction and low quality of knowledge base caused by the existing single technical path, and laying a solid data foundation for subsequent high-precision identification.

[0025] 3. The "dual-model parallel identification" architecture designed in this application introduces a cross-validation mechanism into the artificial intelligence system. By comparing the conclusions of a generative model, namely the master discriminant model (M-LLM), and a discriminant model, namely the validation model (CLIP), it can effectively identify and filter out the "illusions" and misjudgments that may occur in a single model, greatly improving the reliability and robustness of the identification results, enabling it to meet the stringent requirements of high-risk application scenarios such as drug quality control.

[0026] 4. This application significantly reduces the cost and technical threshold of system development, deployment and continuous iteration by using an automated knowledge base construction process, combining the synthesis of scarce doping scenario training data using a conditional diffusion model, and employing LoRA and other parameter fine-tuning techniques to fine-tune the dual models.

[0027] 5. The TCM identification results output by this application are no longer limited to outputting a single medicinal material name label, but can automatically generate a comprehensive, standardized, and structured identification report comparable to that of human experts. This provides front-line pharmaceutical quality control personnel with comprehensive decision support information, realizing a significant value leap from "identification tool" to "intelligent decision assistant". Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is the overall flowchart of the Chinese herbal medicine identification method based on a large model in this application; Figure 2 This is a flowchart of a traditional Chinese medicine identification method based on a large model, as described in an embodiment of this application. Figure 3 This is a schematic diagram of the dual-model parallel discrimination principle in the embodiments of this application; Figure 4 This is a flowchart illustrating the core decision-making process in the embodiments of this application; Figure 5 A bar chart showing the experimental results comparing the performance of different models; Figure 6 This is a technical architecture diagram of the traditional Chinese medicine identification system based on a large model in the embodiments of this application. Detailed Implementation To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] This application provides a method, system, electronic device, and storage medium for identifying traditional Chinese medicine (TCM) based on a large model, which at least solves the problems of poor accuracy and low reliability in existing TCM identification technologies. The core of this method lies in combining a generative master identification model responsible for generating open-ended identification hypotheses ("what is it, why") with a discriminative verification model responsible for providing closed-set quantitative verification ("is it, how similar is it?"). Through parallel processing and mandatory cross-validation, the final output results are ensured to have both high accuracy and high reliability.

[0031] It should be noted that the large-scale model-based TCM identification technology proposed in this application is applicable to the identification of TCM and processed medicinal materials in various application scenarios. For example, it can be used by medical institutions, pharmacies, and TCM manufacturers for batch acceptance of medicinal materials; by drug testing institutes and their quality supervision departments for quality control of TCM materials; and by the general public or TCM identification enthusiasts for identification. In other words, TCM acceptance equipment and medicinal material identification apps developed based on the large-scale model-based TCM identification technology proposed in this application all fall within the technical scope of this application. To facilitate the explanation of the technical solution of this application, the following embodiments will use the example of obtaining the final identification result after a user submits a TCM identification request to illustrate the implementation process of the technical solution of this application in detail. However, this example of a specific application scenario is not intended to limit the scope of protection of this application.

[0032] Example 1: Firstly, this invention proposes a method for identifying traditional Chinese medicine based on a large-scale model, see [link to relevant documentation]. Figure 1 The method includes: S1. Construct a structured knowledge base for traditional Chinese medicine; S2. Based on an asymmetric dual model consisting of a generative master identification model and a discriminative verification model, parallel identification and cross-validation of Chinese medicinal materials are performed. Among them, the generative master identification model is used to generate preliminary identification results based on the image of the Chinese medicinal material to be identified, and the discriminative verification model is used to obtain a list of candidate results with quantifiable confidence scores based on the image of the Chinese medicinal material to be identified and the structured Chinese medicinal knowledge base. S3. Compare the preliminary identification results with the candidate results corresponding to the highest confidence score in the candidate result list. If and only if the preliminary identification results and the candidate results corresponding to the highest confidence score are consistent and the confidence score is higher than the preset verification threshold, the identification is confirmed as successful and the final identification result of the Chinese medicinal materials is output.

[0033] The following is in conjunction with the appendix Figures 1-6 The following details the implementation process of an embodiment of the present invention, including explanations of the specific steps S1-S3.

[0034] S1. Construct a structured knowledge base for traditional Chinese medicine.

[0035] To address the problems of incomplete information extraction and low knowledge base quality in existing methods of constructing traditional Chinese medicine (TCM) knowledge bases due to reliance on a single information source (plain text or plain image), which in turn leads to low accuracy and reliability in visual identification of TCM, this embodiment first proposes a dual-pathway, multimodal automated knowledge base construction method. For example... Figure 2 As shown, specifically: S11. Acquisition and preprocessing of original Chinese medicine knowledge data.

[0036] Authoritative and professional data on traditional Chinese medicine (TCM) are acquired through web scraping or manual downloading, including but not limited to PDF versions of books related to TCM herbs. Supplementary materials such as relevant internet web pages and WeChat public account articles are also collected. To ensure consistent data source formats, all non-PDF content (such as HTML web pages) is converted into standardized PDF documents using automated browser scripts (such as Selenium) that utilize the browser's "print to PDF" function, and then stored in the original document database.

[0037] S12. Based on the document parsing framework, perform document parsing on the preprocessed original Chinese medicine knowledge data.

[0038] To separate text and images from complex PDF documents, an advanced document parsing framework is used to perform in-depth preprocessing on professional PDF documents related to Chinese medicinal materials, accurately parsing them into structured Markdown text and a collection of associated independent image files.

[0039] Preferably, this embodiment uses the Dolphin open-source software framework to parse the aforementioned original traditional Chinese medicine knowledge data. Dolphin employs a novel "analyze-then-parse" two-stage processing paradigm, which is particularly suitable for complex documents with mixed text and images processed in this embodiment. The specific steps of this process are as follows: Standardized PDF documents are input into the Dolphin system. In the first stage (analysis stage), Dolphin performs comprehensive page-level layout analysis, using its built-in visual language model to identify various elements in the document, such as text paragraphs, headings, tables, images, formulas, etc., and predict their natural reading order. In the second stage (parsing stage), based on the analysis results, Dolphin converts the entire PDF document into a structured Markdown text file. During this process, all images in the document are automatically identified, cropped, and saved as independent .jpg files. Importantly, the Markdown text retains the relative position information or citation markers of these images in the original text. Finally, a Markdown file and a series of .jpg images are produced as ideal, decoupled input data for subsequent dual-path parallel information extraction.

[0040] S13. Information extraction processing is performed on the parsed data through two parallel information extraction paths.

[0041] To maximize the extraction of all information from the original Chinese medicine knowledge data documents, two parallel and complementary information extraction pathways are used to process the parsed data. These pathways include text-based structured information extraction (Pathway A) and multimodal large-scale model-based information extraction (Pathway B). Pathway A utilizes a Large Language Model (LLM) to semantically segment and process the Markdown text content. Through carefully designed prompts, it automatically extracts key information such as the name, origin, properties, and indications of the medicines described in the text and structures it into preliminary JSON format data. Pathway B addresses non-textual information that Pathway A cannot process, utilizing a cutting-edge multimodal large-scale language model (M-LLM). Specifically: Pathway A (Text-Based Structured Information Extraction): This path focuses on efficiently processing plain text information. First, the Markdown text content generated by Dolphin is chunked according to a preset character length (e.g., every 1000 characters) to accommodate the context window limitations of the large language model. Next, for each text chunk, a high-performance large language model (e.g., a locally deployed Qwen-2.5-VL-72B model) is called via API, with carefully designed structured prompts input. Finally, the JSON objects returned by the model are collected and parsed to form preliminary structured data.

[0042] The structured prompts include, but are not limited to, the following format and content: "You are a senior pharmaceutical expert and a data extraction robot. Your task is to extract information about traditional Chinese medicine from the following text snippet, strictly adhering to the requirements. Please extract the following fields: 'Name,' 'Source,' 'Properties,' 'Indications,' 'Dosage and Administration,' 'Precautions,' 'Identification Points,' 'Identification Methods Other than Appearance,' and 'Whether there is adulteration or counterfeiting.' Output must be in strict JSON format, where the key is the field name and the value is the corresponding content extracted from the text. If a field's content does not exist in the current text snippet, its corresponding value must be null. Do not output any explanatory text; only output the JSON object." It should be noted that Path A can quickly and accurately process most of the text content in a document, while Path B is used to process non-text information that Path A cannot process.

[0043] Pathway B (Information Extraction Based on Multimodal Large Model): This path aims to address the inherent limitations of Pathway A, namely its inability to extract information from images, scanned tables, and charts. It is crucial for ensuring the integrity of the knowledge base. Specifically: The original PDF document is segmented by page number (e.g., 10 pages per processing unit). For each segment, all page images (or corresponding .jpg images extracted from Dolphin) are used as image input, along with the original text or Markdown text of that segment as text input, and together they are fed into a top-level multimodal large language model.

[0044] Using cue words similar to those in pathway A, the Qwen-2.5-VL-72B model is instructed to extract information by combining its understanding of both images and text. Because the model can directly "see" and understand image content, it can extract key information that pathway A completely ignores. For example, it can extract morphological descriptions such as "cambium rings" from the annotations of a cross-sectional diagram of medicinal materials, or extract structured data from a complex comparison table of identification points.

[0045] In this embodiment, the preferred multimodal large language model is the Qwen-2.5-VL-72B model, which possesses excellent text and image understanding capabilities. In the latest technical evaluations, the Qwen-2.5-VL-72B model demonstrates industry-leading performance in document understanding, chart analysis, and structured data extraction. By directly using the original page images of the document or the extracted image files as input, the model can "read" and understand the content of tables, charts, and images, thereby extracting key identification information that cannot be obtained through a plain text approach.

[0046] S14. Based on the large language model, perform data fusion and verification on the data after information extraction and processing to construct a structured Chinese medicine knowledge base.

[0047] To integrate the results of two information extraction pathways and ensure the accuracy of the final data, this embodiment introduces an intelligent data fusion step. A large language model is used to intelligently fuse and validate the JSON data extracted from the two pathways. After comparison, merging, deduplication, and information supplementation, the large language model ultimately generates a high-quality structured traditional Chinese medicine knowledge base with information completeness far exceeding that of a single technical path. Specifically: For the same traditional Chinese medicine (TCM) entry, two JSON datasets extracted from pathway A and pathway B are simultaneously input as context into a flagship-level large language model with powerful logical reasoning capabilities, such as GPT-4o. Through prompt word engineering, this model is instructed to act as a rigorous "data review and integration expert," with tasks including, but not limited to: 1) Merging information: merging non-repeating information from the two JSON datasets into a new JSON dataset; 2) Supplementing and refining: supplementing the macroscopic morphological description extracted from pathway A with details extracted from the image of pathway B (such as "the surface has wart-like protrusions"); 3) Deduplication and validation: identifying and removing duplicate or contradictory information, selecting more reliable and detailed descriptions; 4) Formatted output: generating a final, most complete, and validated JSON record. The final generated JSON data is stored in a structured TCM knowledge base.

[0048] It should be noted that this data fusion step ensures the completeness and accuracy of the knowledge base information, and significantly improves the quality of the automated knowledge base construction by simulating the review process of human experts.

[0049] S2. Based on an asymmetric dual model consisting of a generative master identification model and a discriminative verification model, parallel identification and cross-validation of Chinese medicinal materials are performed. The generative master identification model is used to generate preliminary identification results based on the image of the Chinese medicinal material to be identified, and the discriminative verification model is used to obtain a list of candidate results with quantifiable confidence scores based on the image of the Chinese medicinal material to be identified and the structured Chinese medicinal knowledge base.

[0050] To ensure the high reliability of the identification results, such as Figure 3 As shown in the figure, this embodiment designs a dual-model parallel identification and cross-validation algorithm. The images of Chinese medicines to be identified are fed into the main identification model and the validation model in parallel. The preliminary identification result is obtained based on the main identification model, and the candidate result list with quantifiable confidence scores is obtained based on the validation model.

[0051] During the inference phase, the images of the Chinese medicinal herbs uploaded by the user are fed in parallel into two independent models. These two independent but complementary models are the main identification model and the validation model, respectively: The primary identification model is a multimodal large language model responsible for deep analysis of Chinese herbal medicine images, generating detailed, expert-interactive preliminary identification results. Its output includes not only the name of the medicinal material but also rich content such as morphological descriptions, key identification features, and quality assessments. In one embodiment, the primary identification model is preferably Qwen-2.5-VL-72B.

[0052] The validation model is a Contrastive Language-Image Pre-training (CLIP) model. This model is responsible for encoding the visual features of the input image into embedding vectors and performing high-precision comparisons with the text embedding vectors of all standard medicinal material names in the structured TCM knowledge base constructed above. Finally, it outputs a list of the top N candidate results with quantifiable confidence scores to quantitatively validate the text hypothesis.

[0053] It should be noted that the proposed primary identification model and verification model play asymmetric and complementary roles in this embodiment, simulating the "bold hypothesis, careful verification" process of human expert identification. The primary identification model (Qwen-2.5-VL-72B), as the "hypotheser," has the advantage of observing images like an expert and comprehensively describing its observations in natural language, including the color, texture, and cross-sectional features of the medicinal materials, and providing a comprehensive preliminary judgment based on these observations. This is an open-ended, divergent reasoning. The verification model (CLIP), as the "verifier," rigidly compares the visual features of the image with the textual concepts of all standard medicinal materials in the knowledge base, providing a convergent quantitative ranking within a closed set by calculating the cosine similarity in the vector space. The output of the primary identification model provides a target for subsequent verification, while the quantitative results of the verification model provide a solid barrier against the "illusion" risk of the primary identification model. Both are indispensable and together constitute the core of the high-reliability identification in this embodiment.

[0054] In one embodiment, to enhance the generalization ability and robustness of the dual-model approach in real-world complex application environments, all images obtained from the image knowledge base (i.e., the structured traditional Chinese medicine knowledge base) undergo a robust data augmentation pipeline before being used for training. Specifically, data augmentation includes, but is not limited to, the following processing of the images obtained from the image knowledge base: 1) Conventional Transformations. A series of conventional geometric and color transformations are applied, including random rotation, scaling, cropping within a certain range, as well as random adjustment of the image's brightness, contrast, saturation, and hue, to simulate different shooting angles, distances, and lighting conditions.

[0055] 2) Background Replacement. This embodiment uses the Segment Anything Model (SAM) for high-precision foreground segmentation of the medicinal herb image. As a powerful basic model, SAM has excellent zero-shot segmentation capabilities, requiring no additional training for traditional Chinese medicine data to accurately separate the main subject (foreground) of the medicinal herb from its original shooting background. The segmented foreground of the medicinal herb is then randomly synthesized with a pre-built background image library containing hundreds of real-world backgrounds (such as pharmacy counters of different materials, stainless steel inspection tables, burlap sacks, wooden tabletops, white background boards, etc.).

[0056] It should be noted that this step is crucial for dealing with background interference. This step forces the model to focus on learning the inherent and stable identification features of the medicinal material itself (such as texture, color, and shape), while ignoring variable and irrelevant background information, thereby significantly improving its performance in practical applications.

[0057] 3) Generation of synthetic data for adulterated scenarios. Collecting large quantities of adulterated samples of traditional Chinese medicine in a systematic manner in the real world is a major bottleneck for model training. This embodiment employs a Conditional Latent Diffusion Model to programmatically generate massive amounts of photorealistic training data for adulterated scenarios. Preferably, the Conditional Latent Diffusion Model includes, but is not limited to, a Stable DiffusionXL model fine-tuned on a refined dataset of medicinal herb images.

[0058] The content of the generated images can be precisely controlled through structured prompt engineering. For example, the following prompt can be input into the model: "A high-resolution photograph of Chinese medicinal herbs, mainly consisting of 85% genuine Astragalus membranaceus slices, randomly mixed with 15% of similar-looking counterfeit Caragana root slices. All slices are randomly scattered in a white ceramic dish, photographed using standard laboratory top lighting and a macro lens, with clear details and photorealistic quality." For each synthesized image generated, the system will synchronously create a corresponding structured label, for example: {"components": [{"name": "Astragalus membranaceus", "ratio": 0.85}, {"name": "Caragana korshinskii root", "ratio":0.15}], "is_adulterated": true}.

[0059] Adding these massive amounts of "synthetic image-label" data pairs to the training dataset can effectively teach the master discrimination model (Qwen-2.5-VL-72B) not only to identify the presence of multiple components, but also to accurately describe the doped substances and their proportions.

[0060] In one embodiment, in order to reduce the reliance on large-scale computing resources (such as multi-GPU clusters) and improve model iteration efficiency, a parameter-efficient fine-tuning technique is used to fine-tune the aforementioned master identification model and verification model.

[0061] Fine-tuning of the master discrimination model (Qwen-2.5-VL-72B): Supervised fine-tuning (SFT) was performed using a carefully constructed hybrid dataset that included: text-image pairs constructed from a knowledge base, the aforementioned synthetic mixed-scene data, and dialogue-formatted data simulating user interactions.

[0062] Low-Rank Adaptation (LoRA) was employed to fine-tune the master discriminator model (Qwen-2.5-VL-72B). The core idea of ​​LoRA is to freeze the vast majority (over 99%) of the weights in the pre-trained model, injecting only small, trainable low-rank matrices into key parts of the model (such as the attention module in the Transformer layer). During training, only the parameters of these newly added low-rank matrices are updated. This method can reduce the GPU memory requirements for training by several times, making it possible to perform domain-knowledge-based fine-tuning of ultra-large 72B models on a single or small number of consumer / professional GPUs.

[0063] Validation Model (CLIP) Fine-tuning: Choose a high-performance pre-trained CLIP model as a base, such as ViT-L / 14 from the OpenCLIP project.

[0064] Fine-tuning is performed using contrastive learning on data pairs consisting of (images of medicinal materials, text of standard medicinal material names). The objective function of training is to maximize the cosine similarity of matching image-text pairs in the embedding space, while minimizing the similarity of mismatched image-text pairs.

[0065] After fine-tuning, the CLIP model became a highly specialized "Chinese medicine validator". Its embedding space can accurately reflect the semantic relationship between the visual characteristics of Chinese medicinal materials and their standard names, providing a reliable quantitative basis for subsequent cross-validation.

[0066] S3. Compare the preliminary identification results with the candidate results corresponding to the highest confidence score in the candidate result list. If and only if the preliminary identification results and the candidate results corresponding to the highest confidence score are consistent and the confidence score is higher than the preset verification threshold, the identification is confirmed as successful and the final identification result of the Chinese medicinal materials is output.

[0067] The system compares and arbitrates the outputs of the two models. Only when the judgment of the primary identification model matches the highest confidence result of the verification model, and the confidence score is higher than a preset threshold, will the system output the final identification conclusion. If the two results are inconsistent or the confidence score is low, a "fuzziness alert" is issued, prompting the user to conduct further manual verification, thereby greatly improving the security and reliability of the system in practical applications.

[0068] In one embodiment, a structured TCM identification report is automatically generated to directly and intuitively present the final identification conclusion of TCM.

[0069] Based on the highly reliable results verified by the dual-model identification algorithm, the system automatically retrieves all information related to the medicinal material from the structured knowledge base and integrates the real-time analysis conclusions of the master identification model. Finally, the system automatically generates a structured auxiliary identification report for traditional Chinese medicine quality. This report is comprehensive, covering the name, source, properties, indications, key identification points, possible adulteration situations and their identification methods, and can indicate the source of information. This function realizes a paradigm shift from providing "simple classification labels" to issuing "professional consultation opinions," providing users with direct, usable, and professional decision support.

[0070] This completes the entire process of the large-scale model-based method for identifying traditional Chinese medicine proposed in this embodiment and its preferred embodiments.

[0071] The following example uses a real-world application scenario where a user uploads an image of the Chinese herb to be identified (Angelica sinensis image) and requests identification, and then obtains the identification result. This example will be used to explain the specific implementation process of the Chinese herb identification method based on a large model proposed in this embodiment and its preferred embodiments.

[0072] 1) Dual-model parallel inference.

[0073] When a user uploads an image of a Chinese herbal medicine to be identified via the client, the image data is sent to the backend inference engine and distributed to two parallel processing pathways: Pathway 1 (Primary Identification Model Inference): The image of the Chinese herbal medicine to be identified is input into the finely tuned Qwen-2.5-VL-72B model. The primary identification model performs a forward propagation, generating a detailed natural language description. For example: "Based on image analysis, this sample is preliminarily identified as 'Angelica sinensis' slices. Its surface is yellowish-brown, the cross-section is yellowish-white, and there are brown oil spots, which matches the description of Angelica sinensis in a certain professional literature. The sample slices are of uniform thickness, with no obvious signs of mold, indicating good quality." The system then uses rule or intent recognition to parse the preferred herbal name "Angelica sinensis" from this text.

[0074] Pathway 2 (Verification Model Inference): The image of the Chinese herbal medicine to be identified is input into a finely tuned CLIP model, which encodes it into a high-dimensional image embedding vector. This vector is fed into a vector database, where batch cosine similarity calculations are performed against the text embedding vectors of hundreds of standard medicinal material names pre-calculated and stored in a structured Chinese herbal medicine knowledge base. The database returns a candidate list sorted from high to low similarity scores, for example: [("Angelica sinensis", 0.95), ("Angelica dahurica", 0.78), ("Angelica dahurica", 0.65),...].

[0075] (ii) Confidence assessment and core decision-making.

[0076] To obtain more reliable and interpretable confidence scores and implement rigorous decision-making logic, perform the following steps: Confidence calibration: Directly using the cosine similarity within the range of [-1, 1] as probability is inaccurate. In this embodiment, the temperature-scaled softmax function is used to calibrate the cosine similarity score (sim) output by the CLIP model to convert it into a more statistically meaningful confidence probability (P). The calculation formula is: P(i) = \frac{\exp(\text{sim}(i) / T)}{\sum_{j} \exp(\text{sim}(j) / T)} It can also be expressed as: Where: represents the confidence probability of the i-th candidate medicinal material. P(i) is a value between 0 and 1, and the sum of the P values of all candidate medicinal materials is 1; i represents the index of the candidate medicinal material; for example, in the candidate list [("Angelica sinensis", 0.95), ("Levisticum officinale", 0.78), ("Angelica dahurica", 0.65),...], the index i of "Angelica sinensis" can be 1, the index i of "Levisticum officinale" can be 2, and so on.

[0077] sim(i) represents the cosine similarity score between the image to be identified and the i-th candidate medicinal material calculated by the CLIP model (verification model). The higher this value, the more similar the model considers the image and the name of the medicinal material in terms of features. Its original range is between [-1, 1].

[0078] T represents a hyperparameter called temperature, which is a key parameter for adjusting the smoothness of the probability distribution. When the value of T is low, it will make the probability distribution more "sharp", that is, magnify the gap between the high score and the second-high score options, making the model more confident in the highest-score option. When T the value is high, it will make the probability distribution smoother, that is, narrow the probability gap between each option. T The value of is determined by experimental tuning on an independent validation set to make the output probability distribution best reflect the true credibility.

[0079] j represents the index variable that traverses all candidate results. According to the standard definition of the softmax function, the role of j here is to traverse each term in the sum in the denominator. That is, if there are N medicinal materials in the candidate list, j It will iterate from 1 to N to ensure that the denominator is the normalization factor of the sum of the scores of all candidates.

[0080] The summation represents the addition of all candidate medicinal materials (traversed by index j) after calculation using exp(sim(j) / T). This sum is a normalization constant used to ensure that the sum of all calculated P(i) equals 1, thus forming an effective probability distribution.

[0081] Core Decision: The decision-making process is as follows: Figure 4 As shown, this includes consistency verification and similarity risk assessment.

[0082] Consistency check: Compare the preferred name resolved by the master identification model (pathway 1 result) with the Top-1 candidate name after calibration by the validation model (pathway 2 result). If they are inconsistent, it is immediately judged as "result ambiguous". At this time, an alarm message will be generated, which reads: "The system has detected a conflict between the results of two independent analysis models and cannot give a highly reliable identification conclusion. It is recommended that you submit this sample to a pharmacology expert for manual review." If they are consistent, proceed to the next step of confidence check.

[0083] Similarity Risk Assessment: Check whether the confidence probability of the Top-2 candidates in the validation model exceeds a preset risk threshold (e.g., P(Top-2) > 0.2). If it exceeds this threshold, it indicates that the sample is highly similar in appearance to another medicinal material, posing a risk of confusion. The system will attach a clear warning message to the final report: "Note: This sample is similar to '..." "If the appearances are highly similar, please carefully distinguish them in conjunction with other identification points (such as smell, physicochemical identification) to avoid confusion." If the confidence probability of the Top-2 candidates does not exceed this threshold, it is considered a successful identification with high confidence.

[0084] (iii) Automated report generation After passing all the above verifications and being determined as a highly reliable identification, the automated report generation process is initiated: Using the identified medicinal material name (such as "Angelica sinensis") as a unique identifier, a precise search is performed in a structured knowledge base to obtain all related structured information (name, source, properties, indications, dosage, precautions, identification points, adulteration, etc.).

[0085] The authoritative, static information retrieved from the knowledge base is intelligently integrated with the real-time, dynamic analysis conclusions generated by the main identification model (such as "the sample surface has xxx characteristics, which meet the superior product standard") and the quantitative confidence data given by the verification model.

[0086] Finally, the system calls a report generation engine (such as a template-based PDF rendering library) to populate the integrated information into a preset professional report template, generating a well-formatted PDF document containing both text and images for auxiliary identification of traditional Chinese medicine quality. This document is then returned to the client for users to view, download, and archive. The report clearly indicates the source of key information (such as morphological descriptions), greatly enhancing its authority and practical value.

[0087] To objectively and quantitatively evaluate the effectiveness and superiority of the "Dual-Model-Based Method for Identifying Traditional Chinese Medicine" (hereinafter referred to as "this invention") proposed in this embodiment, a set of comparative experiments was designed. The aim was to compare the method of this embodiment with existing mainstream technologies (such as CNN-based classifiers and general multimodal large models) on multiple industry standard evaluation metrics, particularly their performance when handling easily confused or mixed complex samples. See Table 1 and Appendix below. Figure 5 The method proposed in this embodiment is compared with existing mainstream technologies.

[0088] Table 1: Comparison and Analysis of the Invention and the Prior Art Based on Table 1 and Appendix above Figure 5 It is evident that the technology presented in this application surpasses existing related technologies in all aspects, including knowledge base construction, specific identification, identification result output, and result reliability. Overall, the method achieves higher accuracy and reliability in the identification of traditional Chinese medicine compared to existing technologies.

[0089] Example 2: Secondly, this invention also provides a traditional Chinese medicine identification system based on a large model, see [link to relevant documentation]. Figure 6 The system mainly includes: The knowledge base construction module is configured to build a structured traditional Chinese medicine knowledge base; The Chinese medicine identification module is configured to identify Chinese medicinal materials based on a dual-model parallel identification and cross-validation approach; wherein, the dual models include: a main identification model for generating preliminary identification results based on the image of the Chinese medicinal material to be identified, and a validation model for obtaining a list of candidate results with quantifiable confidence scores based on the image of the Chinese medicinal material to be identified and the structured Chinese medicine knowledge base; The identification result output module is configured to compare the preliminary identification result with the candidate result corresponding to the highest confidence score. When the preliminary identification result and the candidate result corresponding to the highest confidence score are consistent and the confidence score is higher than the preset threshold, the identification result of the Chinese medicinal material is output.

[0090] In practical implementation, this embodiment proposes a traditional Chinese medicine identification system based on a large model, whose system architecture includes three parts: a client, a backend server, and a database cluster. Specifically: Client: An application that can be deployed on desktop operating systems (such as Windows, macOS) or mobile devices (such as iOS, Android). This client provides users with a user interface for image capture / uploading, text query input, identification report visualization, and interactive dialogue with the backend model.

[0091] Backend Server: A server cluster deployed in the cloud (such as Alibaba Cloud, AWS) or a local data center, hosting the core computing and logic of the system. During runtime, it executes the steps of the large-model-based traditional Chinese medicine identification method described in Embodiment 1 and its preferred embodiments, and mainly includes the following functional modules: The knowledge base construction module is configured to build a structured Chinese medicine knowledge base based on the acquired and preprocessed raw Chinese medicine knowledge data.

[0092] The model training and augmentation module is configured to augment the model training data and to optimize the training of the dual models using the augmented data.

[0093] The Inference Engine Module, also known as the Chinese Medicine Identification Module, is configured to identify Chinese medicinal materials using a dual-model parallel identification and cross-validation approach.

[0094] The report generation module, also known as the identification result output module, is used to integrate information and generate a PDF report.

[0095] Database Cluster: Raw Document Repository: Uses object storage services (such as OSS) to store unstructured raw data such as collected PDFs and web archives; Image Knowledge Base: Uses a distributed file system to store high-quality image files extracted from literature and associated with each Chinese medicine entry.

[0096] Structured Knowledge Base: Uses a document database (such as MongoDB) to store JSON-formatted information on traditional Chinese medicine that has been extracted, fused, and validated using a dual-pathway (pathway A and path B) method.

[0097] Vector Database: This uses a dedicated vector database (such as Milvus or FAISS) to efficiently store and retrieve the text embedding vectors of medicinal herb names for CLIP model validation.

[0098] It is understood that the large-model-based Chinese medicine identification system provided in this embodiment of the invention corresponds to the large-model-based Chinese medicine identification method described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the large-model-based Chinese medicine identification method, and will not be repeated here.

[0099] Example 3: Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the large-model-based method for identifying traditional Chinese medicine as described in any of the above embodiments and their preferred embodiments. The method mainly includes: S1. Construct a structured knowledge base for traditional Chinese medicine; S2. Identify Chinese medicinal materials based on a dual-model parallel identification and cross-validation method, wherein the dual models include: a master identification model for generating preliminary identification results and a validation model for outputting a list of candidate results with quantifiable confidence scores; S3. Compare the preliminary identification results with the candidate results corresponding to the highest confidence score. When the preliminary identification results and the candidate results corresponding to the highest confidence score are consistent and the confidence score is higher than the preset threshold, output the identification results of Chinese medicinal materials.

[0100] It is understood that the electronic device for identifying traditional Chinese medicine based on a large model provided in this embodiment of the invention corresponds to the above-mentioned method and system for identifying traditional Chinese medicine based on a large model. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the method and system for identifying traditional Chinese medicine based on a large model, and will not be repeated here.

[0101] Example 4: Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the large-model-based method for identifying traditional Chinese medicine as described in any of the above embodiments and their preferred embodiments, the method comprising: S1. Construct a structured knowledge base for traditional Chinese medicine; S2. Identify Chinese medicinal materials based on a dual-model parallel identification and cross-validation method, wherein the dual models include: a master identification model for generating preliminary identification results and a validation model for outputting a list of candidate results with quantifiable confidence scores; S3. Compare the preliminary identification results with the candidate results corresponding to the highest confidence score. When the preliminary identification results and the candidate results corresponding to the highest confidence score are consistent and the confidence score is higher than the preset threshold, output the identification results of Chinese medicinal materials.

[0102] It is understood that the large-model-based Chinese medicine identification storage medium provided in this embodiment of the invention corresponds to the large-model-based Chinese medicine identification method and system described above. The explanations, examples, and beneficial effects of its contents can be referred to the corresponding contents in the large-model-based Chinese medicine identification method and system, and will not be repeated here.

Claims

1. A method for identifying traditional Chinese medicine based on a large model, characterized in that, The method includes: Constructing a structured knowledge base for traditional Chinese medicine; Asymmetric dual-model consisting of a generative master identification model and a discriminative verification model is used to perform parallel identification and cross-validation of Chinese medicinal materials. The generative master identification model is used to generate preliminary identification results based on the image of the Chinese medicinal material to be identified, and the discriminative verification model is used to obtain a list of candidate results with quantifiable confidence scores based on the image of the Chinese medicinal material to be identified and the structured Chinese medicinal knowledge base. The preliminary identification result is compared with the candidate result corresponding to the highest confidence score in the candidate result list. If the preliminary identification result and the candidate result corresponding to the highest confidence score are consistent and the confidence score is higher than the preset verification threshold, the identification is confirmed as successful and the final identification result of Chinese medicinal materials is output.

2. The method as described in claim 1, characterized in that, Based on the image of the Chinese herbal medicine to be identified and the structured Chinese herbal medicine knowledge base, the validation model obtains a list of candidate results with quantifiable confidence scores, including: The visual features of the Chinese herbal medicine image to be identified are encoded into an embedding vector, and compared with the text embedding vectors of all standard medicinal material names in the structured Chinese herbal medicine knowledge base, and a list of candidate results with quantifiable confidence scores is output.

3. The method as described in claim 1, characterized in that, The construction of a structured traditional Chinese medicine knowledge base includes: Acquire and preprocess raw Chinese medicine knowledge data; The preprocessed original Chinese medicine knowledge data is parsed using a document parsing framework. The parsed data is processed by extracting information through two parallel information extraction pathways; the two parallel information extraction pathways include text-based structured information extraction and multimodal large model-based information extraction. Based on a large language model, data fusion and verification are performed on the data after information extraction and processing to construct a structured Chinese medicine knowledge base.

4. The method as described in claim 1, characterized in that, The method further includes: The dual model is trained using data augmentation.

5. The method as described in claim 4, characterized in that, The data augmentation includes processing the images of Chinese herbs in the structured Chinese herbal medicine knowledge base as follows: Randomly rotate, scale, and crop images of traditional Chinese medicine within a certain range, and perform geometric and color transformations; and / or The SAM image segmentation model is used to perform high-precision foreground segmentation on images of traditional Chinese medicine; and / or A large amount of training data on doping scenarios related to images of traditional Chinese medicine was generated using a conditional latent diffusion model.

6. The method as described in claim 1, characterized in that, The method further includes: Fine-tuning is performed on the primary identification model and the verification model; the fine-tuning includes: A hybrid dataset is constructed to perform supervised fine-tuning of the primary identification model, and a low-rank adaptive technique is used to fine-tune the primary identification model. A pre-trained validation model was selected as the basis for comparison and fine-tuning with the validation model.

7. The method as described in claim 1, characterized in that, The primary discrimination model includes a multimodal large language model, which includes Qwen-2.5-VL-72B; the validation model includes a contrastive language-image pre-trained model.

8. A traditional Chinese medicine identification system based on a large model, characterized in that, The system includes: The knowledge base construction module is configured to build a structured traditional Chinese medicine knowledge base; The Chinese medicine identification module is configured to identify Chinese medicinal materials based on a dual-model parallel identification and cross-validation approach; wherein, the dual models include: a main identification model for generating preliminary identification results based on the image of the Chinese medicinal material to be identified, and a validation model for obtaining a list of candidate results with quantifiable confidence scores based on the image of the Chinese medicinal material to be identified and the structured Chinese medicine knowledge base; The identification result output module is configured to compare the preliminary identification result with the candidate result corresponding to the highest confidence score. When the preliminary identification result and the candidate result corresponding to the highest confidence score are consistent and the confidence score is higher than the preset threshold, the identification result of the Chinese medicinal material is output.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the traditional Chinese medicine identification method based on a large model as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the large-model-based method for identifying traditional Chinese medicine as described in any one of claims 1-7.

Images