Image classification method for Chinese medicinal materials

Abstract

This invention relates to the interdisciplinary field of image recognition and traditional Chinese medicine (TCM) identification, specifically disclosing a method for image classification of TCM medicinal materials. The method includes: acquiring and preprocessing high-resolution images of medicinal materials; constructing a TCM knowledge graph based on the "principal, assistant, adjuvant, and guide" principle of TCM compatibility; extracting visual features of candidate regions through a multi-scale feature fusion network; calculating category similarity in a visual-semantic joint embedding space; introducing a causal inference mechanism, combining compatibility logic and occlusion relationships to correct classification confidence; and finally outputting recognition results with location information and prescription role attributes. This invention improves recognition accuracy and system robustness in complex stacked scenarios by integrating knowledge graphs and causal reasoning, supporting prescription compliance verification and the digital application of TCM.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of image recognition and traditional Chinese medicine identification, specifically involving a method for classifying images of traditional Chinese medicine materials. Background Technology

[0002] With the deepening modernization of traditional Chinese medicine (TCM), the automatic identification and classification technology of Chinese medicinal materials plays a crucial role in smart pharmacies, drug traceability, and automated prescription verification. Traditional identification of Chinese medicinal materials mainly relies on pharmacists' sensory experience for manual identification, which is inefficient and lacks the standardization required for large-scale automated distribution and precise supervision. To improve the safety and accuracy of medicinal material distribution, utilizing computer vision technology to automate the classification of Chinese medicinal materials has become a core trend in the industry.

[0003] The image classification method for identifying traditional Chinese medicine (TCM) herbs aims to use deep learning models to extract and compare visual features such as texture, shape, and color of medicinal materials, thereby achieving rapid classification of medicinal material varieties. In practical applications, the system needs to simulate the identification logic of professionals through algorithms to accurately capture the identification points of various medicinal materials from complex backgrounds. This requires the image classification model to not only have basic pixel feature parsing capabilities but also the ability to handle logical connections and integrate prior knowledge in scenarios where multiple objects coexist.

[0004] Existing medicinal herb recognition models mostly focus on matching the visual features of a single target. When faced with complex scenarios where multiple medicinal herbs are randomly stacked, physical occlusion often leads to the loss of key deformation information and texture features, resulting in ambiguous recognition. For easily confused varieties with highly similar appearances, traditional classification algorithms lack in-depth discrimination of microscopic differences, leading to a high false positive rate.

[0005] Current identification schemes fail to effectively integrate the inherent logic of traditional Chinese medicine formulas, cannot utilize the compatibility relationships between medicinal materials for auxiliary reasoning, and lack the ability to causally infer the co-occurrence probability between image targets, resulting in insufficient robustness of the system when facing complex prescription scenarios. These shortcomings collectively limit the accuracy of automatic medicinal material identification technology, forming a pressing technical challenge: the desired image classification method for traditional Chinese medicine medicinal material identification. Summary of the Invention

[0006] The purpose of this invention is to provide a method for identifying and classifying images of traditional Chinese medicine materials, which can effectively solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is: a method for image classification of traditional Chinese medicine materials, comprising the following specific steps:

[0008] Step 1: Collect raw image data containing multiple medicinal herbs stacked together. Use a high-resolution imaging device to obtain color images of the herbs under natural or controlled light conditions, and perform noise reduction and color correction preprocessing on the images.

[0009] Step 2: Construct a knowledge graph based on the "principal, assistant, adjuvant, and guide" principle of traditional Chinese medicine prescriptions. The knowledge graph uses medicinal materials as nodes and compatibility relationships as edges to form a directed graph model with hierarchical structure and semantic constraints.

[0010] Step 3: Extract candidate medicinal material regions from the preprocessed image, and use a multi-scale feature fusion network to encode the local texture, contour and color features of each region to generate a preliminary visual feature vector;

[0011] Step 4: Align the visual feature vector with the node information in the knowledge graph across modalities to establish a visual-semantic joint embedding space, and calculate the similarity score between each candidate region and the known medicinal material category in the visual-semantic joint embedding space;

[0012] Step 5: Introduce a causal inference mechanism to analyze the co-occurrence patterns and occlusion relationships among multiple medicinal material targets in the image. Based on the matching logic in the knowledge graph, determine whether there is a lack of key features due to physical occlusion, and adjust the classification confidence of each candidate region accordingly.

[0013] Step 6: Combine the visual similarity score with the confidence score after causal inference correction, and output the final medicinal material identification result, including the category label of each medicinal material and its location information in the image.

[0014] Preferably, the high-resolution imaging device in step 1 is equipped with an adjustable light source system, which can acquire images under different color temperatures and illuminance conditions to enhance the visibility of details on the surface of the medicinal materials, and eliminate ambient light interference through a white balance correction algorithm to ensure image color consistency.

[0015] Preferably, the knowledge graph construction process in step 2 includes: extracting the compatibility rules of medicinal materials from authoritative Chinese medicine classics and clinical prescription databases, taking the "principal drug" as the core node, and the "assistant drug", "adjuvant drug" and "guide drug" as their child nodes respectively, establishing edge weights based on functional synergy, property and flavor compatibility and toxicity constraints, and forming a structured knowledge system with reasoning ability.

[0016] Preferably, in step 3, the multi-scale feature fusion network consists of multiple parallel convolutional branches, each processing image patches under different receptive fields. The features at each scale are dynamically weighted through a channel attention mechanism, and finally spliced ​​together to generate a unified visual feature vector, so as to take into account both local details and overall morphological expression.

[0017] Preferably, in step 4, the cross-modal alignment adopts a contrastive learning strategy. During the training phase, a large number of labeled samples are used to simultaneously optimize the visual encoder and the semantic embedding module, so that similar medicinal materials are closer in the joint space and dissimilar medicinal materials are farther apart, thereby improving the clarity of the classification boundary.

[0018] Preferably, the causal inference mechanism in step 5 is based on Bayesian network modeling, which uses the co-occurrence frequency of medicinal materials, compatibility and degree of occlusion as observable variables to infer the potential real combination state of medicinal materials, and corrects the initial classification result with posterior probability accordingly, thereby reducing the risk of misjudgment caused by occlusion or confusion.

[0019] Preferably, the final identification result in step 6 includes not only the category of medicinal material, but also its role attribute in the prescription, i.e. whether it is a principal drug, assistant drug, adjuvant drug, or guiding drug. This role attribute is automatically assigned by the node level in the knowledge graph to support subsequent prescription compliance verification.

[0020] Preferably, the method also includes a mechanism for feedback learning of failed identification cases. When the system detects low confidence or contradictory classification results, it automatically triggers a manual review interface and feeds the corrected labels back to the training dataset to continuously optimize model performance.

[0021] Preferably, the knowledge graph supports dynamic updates and can automatically expand the relationships between nodes and edges based on newly released pharmacopoeia standards or clinical research progress, thus maintaining the system's adaptability to the latest TCM knowledge.

[0022] Preferably, the multi-scale feature fusion network is designed to be lightweight during deployment. It compresses the model volume through depthwise separable convolution and channel pruning techniques, making it suitable for edge computing devices and meeting real-time requirements while ensuring recognition accuracy.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. This invention constructs an image classification method for Chinese medicinal materials that integrates computer vision, knowledge graphs of traditional Chinese medicine formulas, and causal inference algorithms. This method possesses prior knowledge guidance and logical reasoning capabilities. It overcomes the problem of missing key features due to physical occlusion in scenarios with multiple stacked medicinal materials.

[0025] 2. This invention improves the ability to distinguish easily confused varieties. By using the "principal, assistant, adjuvant, and guide" matching logic to perform semantic constraints and causal corrections on the identification results, the robustness and reliability of the system in complex prescription environments are enhanced.

[0026] 3. This invention realizes a paradigm shift from pure visual matching to knowledge-driven intelligent recognition, which not only improves the classification accuracy, but also provides reliable technical support for automated prescription review, identification of genuine and counterfeit medicinal materials, and digital inheritance of traditional Chinese medicine. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the overall technical solution architecture according to the present invention;

[0028] Figure 2 This is a logical framework diagram for knowledge graph construction and semantic constraints based on the "ruler-minister-assistant-envoy" matching principle in this invention;

[0029] Figure 3 This is a schematic diagram illustrating the core principle framework for establishing a visual-semantic joint embedding space and performing cross-modal alignment according to the present invention.

[0030] Figure 4 This is a flowchart of the co-occurrence pattern analysis and classification confidence correction based on Bayesian network causal inference according to the present invention;

[0031] Figure 5 This is a schematic diagram illustrating the data flow for extracting visual feature vectors of candidate regions using the multi-scale feature fusion network according to the present invention. Detailed Implementation

[0032] Example 1: Please refer to the appendix Figure 1 To be continued Figure 5 To make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments.

[0033] In the specific implementation of the Chinese herbal medicine image classification method, the entire recognition process is divided into several logically rigorous and interrelated stages.

[0034] For step 1, raw image data containing the stacked state of multiple medicinal herbs is acquired. In this stage, an industrial-grade imaging device equipped with a high-resolution image sensor is used to acquire multi-dimensional images of the medicinal herbs placed on the material support platform. The high-resolution imaging device has an imaging capability of no less than 20 million effective pixels, capable of capturing the fine textures, color differences, and edge contour details of the herb surface. To ensure the stability of image acquisition, the imaging device is mounted on a shock-absorbing bracket, with its optical axis perpendicular to the plane of the material support platform.

[0035] Step 1 includes an adjustable light source system. This system uses an LED array structure and can provide various lighting environments, including cool color temperature, warm color temperature, and natural light color temperature. The light source system is adjusted by a control unit, allowing the illuminance to be adjustable between 1000 lux and 5000 lux. During image acquisition, the system automatically adjusts the incident angle and intensity of the light source based on the reflective characteristics of the medicinal material's surface. For example, for glossy seed-type medicinal materials, the system lowers the incident angle to reduce high-light reflection; for rough-surfaced root and rhizome-type medicinal materials, it enhances scattered light to highlight their epidermal wrinkles.

[0036] After acquiring the color image, the system immediately performs denoising and color correction preprocessing. The denoising process includes applying an improved nonlocal mean filtering algorithm. This algorithm searches for similar pixel blocks within the image and performs a weighted average, removing thermal noise and shot noise generated by the photosensitive element while preserving the edge details of the medicinal materials. The color correction preprocessing uses a standard color chart as a reference.

[0037] Simultaneously or periodically, reference images containing a 24-color standard color chart are acquired while acquiring images of medicinal herbs, and the component offset values ​​of each color patch under the current light source conditions are calculated. A white balance correction algorithm is used to calculate the gain coefficients of the red, green, and blue channels of the entire image, ensuring that the channel values ​​in the white areas of the image are consistent, eliminating the interference of ambient light color temperature on the true color of the medicinal herbs, and ensuring color consistency of the same variety of medicinal herbs at different acquisition times.

[0038] For step 2, a knowledge graph based on the "principal, assistant, adjuvant, and guide" principle of traditional Chinese medicine prescriptions is constructed. This step involves in-depth mining and structured processing of massive amounts of traditional Chinese medicine text data. The construction process of the knowledge graph first extracts medicinal material entities and their relationships from the Chinese Pharmacopoeia, authoritative traditional Chinese medicine classics, and massive clinical prescription databases. The knowledge graph uses medicinal materials as core nodes, and each node is associated with rich attribute information, including but not limited to the name, alternative names, properties, meridians, and efficacy categories of the medicinal material.

[0039] The edges of the knowledge graph represent the compatibility relationships between medicinal materials, forming a directed graph model with hierarchical structure and semantic constraints. The "principal herb" is defined as the highest-level node in the prescription, representing the herb that plays a primary therapeutic role against the main symptom. Below the "principal herb" node, downward edges are established based on the logic of "assistant herbs" assisting the "principal herb" to enhance the therapeutic effect or to address concurrent symptoms. Multi-level associations are then established sequentially based on the logic of "adjuvant herbs" cooperating with the "principal and assistant herbs" to enhance efficacy, eliminate toxicity, or assist in treatment, and the logic of "guide herbs" guiding the medicinal power to the disease site or harmonizing the various herbs.

[0040] Each edge is assigned a weight value, which is determined by the co-occurrence frequency of the medicinal material in historical prescriptions, the tightness of their compatibility, and the synergistic chemical reactions between the drugs. For example, in the knowledge structure of Ma Huang Tang (Ephedra Decoction), Ma Huang (Ephedra) is the principal drug node, and it has a highly weighted synergistic edge with the assistant drug Gui Zhi (Cinnamon Twig). This structured knowledge system provides strong logical support for subsequent visual recognition, enabling the system to possess reasoning abilities similar to those of a traditional Chinese medicine expert.

[0041] For step 3, candidate medicinal herb regions are extracted from the preprocessed image. The system employs an improved selective search algorithm or region proposal network to generate multiple regions of interest (ROIs) in the entire image that may contain a single medicinal herb, based on pixel color similarity, texture correlation, and bounding box compactness. Subsequently, these candidate regions are input into a multi-scale feature fusion network for deep feature extraction.

[0042] The multi-scale feature fusion network consists of multiple parallel convolutional branches. Each branch uses convolutional kernels of different sizes, such as 3×3, 5×5, and 7×7 kernels, to process image patches with different receptive field sizes. Larger receptive fields are used to capture the overall morphological features of the medicinal material, such as the elongated shape of the rhizome or the flattened shape of the leaves; smaller receptive fields focus on extracting local texture features, such as surface hairs, the rings of slices, or small stomata.

[0043] To optimize feature extraction, a channel attention mechanism is introduced into the network. This mechanism obtains global statistical information along the channel dimension by performing global average pooling on the feature map of each channel. Then, two fully connected layers learn the importance weights for each channel. These weights are applied to the original feature map, enabling the network to dynamically weight features at different scales, automatically strengthening key features that contribute to classification while suppressing redundant noise features. The feature vectors extracted from each branch are concatenated along the channel dimension to generate a unified, high-dimensional visual feature vector, achieving a balanced representation of both local details and the overall morphology of the medicinal material.

[0044] For step 4, the visual feature vector is cross-modal aligned with the node information in the knowledge graph. Since visual features belong to a continuous numerical space, while semantic information in the knowledge graph belongs to a discrete symbolic space, the system establishes a common visual-semantic joint embedding space. In this visual-semantic joint embedding space, the vector output by the visual encoder and the vector output by the semantic embedding module have the same dimension.

[0045] The cross-modal alignment employs a contrastive learning strategy. During model training, a database containing massive amounts of labeled samples is used to simultaneously optimize the parameters of the visual encoder and semantic embedding module. For medicinal materials of the same category, the system guides the Euclidean distance between their visual feature vectors and corresponding semantic embedding vectors in the joint space to be minimized; while for medicinal materials of different categories, it guides the vector spacing to be maximized. Specifically, a similarity score is obtained by calculating the dot product of two vectors and then normalizing the results.

[0046] This similarity score reflects the degree of matching between the current candidate region image and a specific category of medicinal materials in the knowledge graph. Due to the introduction of semantic constraints, even when visual features are similar, the system can utilize semantic information such as different properties and functions in the knowledge graph to improve the clarity of classification boundaries and reduce misjudgments caused by visual similarity.

[0047] For step 5, a causal inference mechanism is introduced. In complex scenarios with multiple medicinal herbs stacked together, physical occlusion can lead to the loss of key features of the herbs. The system models this process based on a Bayesian network. The Bayesian network uses the co-occurrence frequency, compatibility, occlusion ratio, and lighting conditions of the herbs as observable variables, and the true category state of the herbs as latent variables.

[0048] When incomplete visual features are detected in a candidate region, the causal inference mechanism analyzes the distribution of medicinal herbs surrounding that region. The system searches the knowledge graph to find the compatibility rules for the identified herbs. For example, if "ephedra" and "cinnamon twig" have been identified with high confidence in the image, the probability of "almond" and "licorice" appearing in the prescription will increase according to the principle of "principal, assistant, adjuvant, and guide".

[0049] The system calculates the logical causal relationship strength of this combination. If the residual features exhibited by a severely occluded area match the color or local edges of an "almond," the causal inference mechanism transforms this "combination necessity" into a correction factor. Based on formulaic logic, the system uses the initial visual classification probability as a prior distribution, combines it with the causal correction probability generated based on a knowledge graph, and calculates the adjusted classification confidence using the posterior probability. This mechanism effectively utilizes the logical relationships between medicinal materials to compensate for missing visual information; even if key identification areas are occluded, correct identification results can be obtained through "logical completion."

[0050] For step 6, the final medicinal herb identification result is output by combining the visual similarity score with the confidence score corrected for causal inference. The final output includes not only the optimal category label for each herb but also its pixel-level location information in the image. Thanks to the introduction of a knowledge graph, the identification result also includes the role attribute of the herb in the current prescription scenario, clearly indicating which herb is the "principal herb" and which are "assistant, adjuvant, or guiding" herbs. This attribute information is automatically assigned by the node hierarchy in the knowledge graph, providing direct data support for subsequent automatic verification of prescription compliance.

[0051] Throughout the system's operation, a feedback learning mechanism for failed identification cases is also included. When the system's highest confidence level falls below a preset threshold, or when the identified herbal combinations severely violate traditional Chinese medicine contraindications, the system automatically triggers a manual review interface. Human experts annotate the controversial images online, and the corrected labels, along with the original image data, are fed back into the training dataset. The system periodically uses this new data to fine-tune the multi-scale feature fusion network and cross-modal alignment module, achieving continuous evolution of model performance.

[0052] To maintain the system's adaptability to the latest TCM knowledge, the knowledge graph supports a dynamic update mechanism. The system can regularly monitor the updates and releases of authoritative pharmacopoeias or the latest clinical research progress. When new medicinal materials are added to the database or existing compatibility relationships are adjusted, the update module automatically adds or removes nodes or modifies the weights of edges in the knowledge graph. This eliminates the need to retrain the entire recognition model from scratch; it can quickly adapt simply by updating the semantic embedding space mapping.

[0053] The multi-scale feature fusion network employs a lightweight design. Before model deployment, standard convolutions are replaced with depthwise separable convolutions, decoupling spatial and channel convolutions and reducing the number of computational parameters. Channel pruning techniques are applied to remove redundant channels in the network that contribute very little to the features. The compressed model size is reduced, enabling it to run smoothly on embedded edge computing devices or mobile terminals. This lightweight design ensures recognition accuracy while meeting the technical requirements for real-time recognition in on-site environments such as pharmacies and clinics.

[0054] In a specific application example, consider a typical stacked scenario containing four medicinal herbs: ephedra, cinnamon twig, apricot kernel, and licorice. The acquisition system first obtains a high-resolution color image of this typical stacked state. In the preprocessing stage, white balance correction ensures that the yellowish-brown color of licorice and the milky white color of apricot kernel are accurately reproduced. In the feature extraction stage, a multi-scale network simultaneously captures the segmented structure of ephedra and the elongated outline of cinnamon twig.

[0055] In the cross-modal alignment space, these visual features were initially matched with the corresponding semantic vectors in the knowledge graph. Due to the large area of ​​ephedra obscuring the almonds, the visual confidence was low. The causal inference module was activated, identifying the presence of ephedra (the principal herb) and cinnamon (an assistant herb) in the image. Based on the classic compatibility logic of "Ma Huang Tang" (Ephedra Decoction), the system automatically increased the category weights of almonds and licorice in the candidate regions. Ultimately, the system not only accurately identified these four herbs but also generated a structured report indicating the location of each herb and pointing out that the combination conformed to the classic pungent-warm exterior-releasing prescription standard.

[0056] In step 3, the data flow within the multi-scale feature fusion network is highly refined. Each convolutional branch generates a series of intermediate feature maps of different dimensions when processing the image. To achieve efficient fusion of deep semantic features and shallow geometric features, the network employs a connection structure similar to a feature pyramid. The high-level abstract features output by the deep network are upsampled to maintain the same spatial resolution as the shallow feature maps, and information is composited by adding them pixel by pixel. This approach ensures that the final generated visual feature vector retains both the macroscopic shape information of the medicinal material and the microscopic texture information that determines species differences.

[0057] In the contrastive learning process of step 4, the selection logic for positive and negative samples is as follows: For any medicinal herb image region, its corresponding correct category label is defined as a positive sample, while other medicinal herb labels that may appear in the same prescription and completely unrelated medicinal herb labels are defined as negative samples. By maximizing the cosine similarity between positive sample pairs and minimizing the similarity between negative sample pairs, the model forms highly clustered clusters in the joint space. Each cluster represents a specific medicinal herb category, and the interval between clusters is the confidence margin for classification. This spatial layout enhances the model's accuracy in distinguishing similar medicinal herbs.

[0058] In the causal inference logic of step 5, the conditional probability table of the Bayesian network is derived from statistics of tens of thousands of historical real prescriptions. When the visual system observes that herb A and herb B appear simultaneously, it calculates the probability increase factor of herb C appearing based on statistical laws.

[0059] If this increase exceeds the uncertainty range given by the visual classifier, the system will intervene and forcibly correct the classification result. This causal intervention can effectively eliminate interference from "false features" caused by overlapping light and shadow. For example, when the shadow of cinnamon twig falls on licorice to form a texture similar to a certain pattern, causal inference will determine that the texture is a non-essential feature based on the logical stability of licorice in the prescription, thus maintaining the correct classification conclusion.

[0060] In the results display stage of step 6, the positioning information is determined in coordinate form. The system calculates the pixel coordinates of the center point of each medicinal material target, as well as the width and height of the bounding rectangle. For medicinal materials with complex shapes, the system also outputs pixel-level mask projections to calculate the area ratio occupied by the medicinal material in the image. This ratio data, combined with the density constant of the medicinal material, can be used to preliminarily estimate the weight ratio of each medicinal material, providing a reference for the review of prescription quantities.

[0061] Example 2: Based on Example 1, this example provides an optimized identification scheme for sliced ​​and processed Chinese medicine in an automated medicine cabinet scenario. In step 1, a high-resolution imaging device is integrated into the mechanical gripper of the automated medicine cabinet. When the gripper removes the medicinal materials from the medicine drawer, the integrated camera captures images of the medicinal materials from multiple angles to address the highly irregular and fragile nature of the processed medicinal materials.

[0062] The knowledge graph in step 2 additionally includes the "processing state" attribute of the medicinal material. Different processing methods alter the visual characteristics of the medicinal material. A chain of associations—"property material - processing method - visual change features"—is established in the knowledge graph. In step 3, during feature extraction, the multi-scale feature fusion network specifically adds a color moment feature extraction branch to capture subtle color shifts that occur after processing.

[0063] In the cross-modal alignment of step 4, the semantic space includes not only the categories of medicinal materials but also descriptors of processing states. This enables the system to distinguish between "raw licorice" and "processed licorice." In the causal inference of step 5, the system infers the required processing state based on the function orientation of the prescription. If the purpose of the prescription is to tonify the spleen, the causal logic increases the confidence of "processed" medicinal materials and corrects the color discrimination bias caused by dim lighting.

[0064] In step 6, the output recognition result is further refined into "medicinal herb name + processing specifications". For example, the output result is "Angelica sinensis (wine-processed Angelica sinensis)", and its complete coverage outline in the image is given. This embodiment is particularly suitable for the digital transformation of traditional Chinese medicine pharmacies, using a vision system to monitor the dispensing process in real time, ensuring that the type and specifications of the dispensed medicine are completely consistent with the prescription.

[0065] Example 3: Based on Example 1, this example details how to use this method to identify the authenticity of medicinal materials and remove impurities in large-scale traditional Chinese medicine warehousing and logistics. In step 1, a conveyor belt-type image acquisition device is used. As the medicinal materials pass through the conveyor belt, a linear industrial camera continuously scans at an extremely high frequency, generating large-format, high-resolution images.

[0066] The knowledge graph in step 2 introduces nodes for "counterfeit and easily confused products." These nodes are connected to genuine medicinal materials through "visual similarity" edges, but differ in semantic attributes such as properties, taste, and efficacy. In step 3, to cope with large-scale data processing, the multi-scale feature fusion network adopts a deeper residual connection structure to extract more discriminative deep semantic features, ensuring the ability to identify even the most subtle counterfeit features.

[0067] In step 4, cross-modal alignment introduces "origin semantics." The same medicinal material from different origins exhibits subtle but stable differences in visual features. By establishing an "origin-visual" mapping in the joint space, the system can achieve preliminary traceability of the medicinal material's origin. In the causal inference of step 5, "batch consistency" is introduced as a causal variable. If the vast majority of medicinal materials in the same batch are identified as originating from a certain region, then for individual medicinal materials with ambiguous features, the system will increase the confidence level of that origin based on batch consistency logic.

[0068] In step 6, the system not only outputs the category of medicinal materials but also marks non-medicinal materials and impurities in the image in real time. Once impurities or counterfeit products are detected, the system will immediately generate an alarm signal and record the real-time coordinates of the impurities on the conveyor belt, triggering the pneumatic sorting device at the back end to physically remove them.

[0069] Example 4: Building upon Example 1, this example proposes a mobile augmented reality (AR)-based TCM teaching recognition scheme. In this scheme, image acquisition in step 1 is performed by a smartphone camera. Due to the performance limitations of smartphone sensors, an image super-resolution reconstruction algorithm is added to the preprocessing stage in step 1. This algorithm uses a deep learning model to convert the low-resolution smartphone preview image into a high-resolution image to complete the details of the medicinal materials.

[0070] The knowledge graph in step 2 is further enriched by adding cultural attribute nodes such as "original appearance of the plant / animal from which the medicinal material originates," "records in ancient books," and "explanations of historical allusions." In step 3, considering the computing power limitations of mobile devices, the multi-scale feature fusion network adopts extreme channel compression and knowledge distillation techniques. The large server-side model guides the training of the small mobile model, enabling the mobile model to achieve feature extraction accuracy close to that of the server-side model.

[0071] In step 4, the alignment result of visual features and semantic information is used to trigger the AR display. When a user points their phone camera at a pile of medicinal herbs, the system overlays detailed information about the herbs onto the phone screen at their physical locations. In step 5, a causal inference mechanism is used to address the issues of camera shake and occlusion during handheld shooting. If the user's finger partially obscures the herbs, the system logically completes the image based on the exposed portion and common teaching herbal combinations, maintaining the stability of the recognition box.

[0072] In step 6, the output is presented as interactive AR tags. Users can click on a tag to access a deep learning page about the medicinal material and view its evolution within the "principal, assistant, adjuvant, and guide" system. This application model facilitates on-site practical identification training for students majoring in traditional Chinese medicine.

[0073] Example 5: Based on Example 1, this example is customized for monitoring the feeding process in a large-scale medicinal herb extract production line. In step 1, the imaging device is installed above the feeding port of the extraction tank. Because the medicinal herbs are in a dynamic falling and overlapping state during the feeding process, the system uses a high frame rate camera for dynamic capture.

[0074] The knowledge graph in step 2 incorporates a "feed ratio constraint" rule. For specific extraction processes, the feeding ratio of different medicinal materials is strictly limited. In the causal inference step 5, the system accumulates the area or estimated weight of each identified medicinal material in real time. If the accumulated amount of a certain medicinal material has reached the upper limit required by the prescription, the causal inference mechanism will automatically lower the recognition probability of the medicinal material in subsequent images and instead focus on finding medicinal material categories that have not yet been fully fed.

[0075] In step 6, the output is integrated with the factory's Manufacturing Execution System (MES). The system generates a composition chart for each batch of materials in real time and automatically compares it with the standard operating procedure (SOP). If an error in the type of material being fed or a serious imbalance in the proportion is detected, the system will automatically cut off the feeding power source to prevent the entire batch of medicinal materials from being wasted due to human error.

[0076] Example 6: Building upon Example 1, this example explores the application of online prescription review in telemedicine and smart pharmacies. In step 1, images are captured by the patient or primary care physician using a dedicated small camera box and uploaded to a cloud server. The camera box provides a standardized, uniform light source.

[0077] The knowledge graph in step 2 is synchronized in real time via the Internet to ensure that the prescription review logic always follows the latest clinical medication guidelines. In step 4, the system introduces a large-scale multimodal pre-trained model, which aligns not only visual and textual information, but also pharmacological experimental data of medicinal materials.

[0078] In the causal inference step 5, the system uses the patient's electronic medical record information as one of the input variables. If the medical record shows that the patient has a "hot" constitution, the system will increase its logical sensitivity to herbs with cold properties when identifying medicinal materials in the image. This three-in-one causal model of "disease-prescription-medicinal materials" makes the identification process more in line with the core ideas of traditional Chinese medicine's syndrome differentiation and treatment.

[0079] In step 6, the output includes a detailed prescription review report. If the identified herbal combinations have incompatibilities, the system will mark the conflicting herbal pairs with red circles on the image and provide risk warnings based on theoretical evidence from the knowledge graph, providing decision support for remote prescription reviewers.

[0080] Example 7: Based on Example 1, this example improves the method for environmental adaptability in the field identification of medicinal materials during the survey of traditional Chinese medicine resources. In step 1, the system takes into account the large fluctuations in natural light intensity in the field and adds adaptive histogram equalization preprocessing, which effectively solves the problem of detail loss caused by overexposure in strong light or excessive darkness in shadows.

[0081] The knowledge graph in step 2 includes Geographic Information System (GIS) attributes, containing information about the medicinal materials' growing environment, such as altitude, latitude, and soil type. In the causal inference in step 5, the system uses the current device's GPS location information as an environmental causal variable. For example, if the location is in a high-altitude area in the south, the system will automatically increase the recognition weight of medicinal material varieties unique to that region, excluding interference from similar varieties from the north.

[0082] In step 6, the identification results include not only the name of the medicinal material but also an evaluation of its distribution density in the current geographical location. Through continuous image acquisition and identification, the system can automatically create a map of the distribution of medicinal resources in the region, providing scientific data support for the protection and development of traditional Chinese medicine resources.

[0083] Example 8: Based on Example 1, this example describes the application of this method in automated medicine vending machines or unmanned traditional Chinese medicine shops. In step 1, the identification system is linked with the inventory management system to monitor the inventory status of medicinal materials on the shelves.

[0084] The knowledge graph in step 2 links inventory codes with price information. In step 5, a causal inference mechanism is used to identify the "expired" or "spoiled" state of the medicinal materials. For example, if an image shows mold spots or an abnormally dull color on the surface of the medicinal materials, the system will determine that the medicinal materials are unsaleable based on the causal variable of their storage time.

[0085] In step 6, the recognition result directly triggers the automatic settlement process. Users simply place the selected medicinal materials on the tray, and the system identifies all varieties, specifications, and quantities at once, and automatically generates a payment QR code.

[0086] Example 9: Based on Example 1, this example expands on the identification of microscopic features in traditional Chinese medicine powders. In step 1, a high-resolution imaging device is coupled with an optical microscope to acquire microscopic images of the medicinal powder.

[0087] The knowledge graph in step 2 contains the microstructural features of the medicinal materials. In step 3, the multi-scale feature fusion network is specifically optimized for the geometric morphology under a microscope. In step 5, causal inference is used to identify adulterants in the powder. If a specific microscopic feature that does not belong to the medicinal material is observed, the causal mechanism calculates its co-occurrence contradiction with the main medicinal material feature to determine whether artificial adulteration exists.

[0088] In step 6, the output is a purity assessment report of the medicinal powder, which details the quantity distribution of various microscopic features, providing a basis for the quality grade assessment of the medicinal powder.

[0089] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for image classification of traditional Chinese medicinal materials, characterized in that, Includes the following steps: Step 1: Collect raw image data containing multiple medicinal herbs stacked together. Use a high-resolution imaging device to obtain color images of the herbs under natural or controlled light conditions, and perform noise reduction and color correction preprocessing on the images. Step 2: Construct a knowledge graph based on the principles of monarch, minister, assistant, and guide in traditional Chinese medicine prescriptions. The knowledge graph uses medicinal materials as nodes and compatibility relationships as edges to form a directed graph model with hierarchical structure and semantic constraints. Step 3: Extract candidate medicinal material regions from the preprocessed image, and use a multi-scale feature fusion network to encode the local texture, contour and color features of each candidate medicinal material region to generate a visual feature vector; Step 4: Align the visual feature vectors with the node information in the knowledge graph across modalities to establish a visual-semantic joint embedding space, and calculate the similarity score between each candidate medicinal material region and the known medicinal material category in the visual-semantic joint embedding space; Step 5: Introduce a causal inference mechanism to analyze the co-occurrence patterns and occlusion relationships among multiple medicinal material targets in the image. Based on the matching logic in the knowledge graph, determine whether there is a lack of key features due to physical occlusion, and adjust the classification confidence of each candidate medicinal material region accordingly. Step 6: Combine the similarity score with the confidence score after causal inference correction, and output the final medicinal material identification result. The medicinal material identification result includes the category label of each medicinal material and its location information in the image.

2. The method for identifying and classifying images of traditional Chinese medicinal materials according to claim 1, characterized in that, Step 1 specifically includes: configuring an imaging device with a photosensitive element to acquire multi-dimensional images of Chinese medicinal materials placed on a material support platform. The imaging device is installed on a support with shock absorption function and its optical axis is perpendicular to the plane of the material support platform. An adjustable light source system provides illumination with different color temperature environments. The control unit adjusts the incident angle and intensity of the light source according to the reflective characteristics of the medicinal material surface. For medicinal materials with a glossy surface, the incident angle is lowered, and for medicinal materials with a rough surface, the scattered light is enhanced. Preprocessing is performed on the acquired color image. The noise reduction process includes applying a nonlocal mean filtering algorithm, which searches for similar pixel blocks in the image and performs a weighted average to remove noise generated by the photosensitive element while preserving the edge details of the medicinal materials. Color correction preprocessing introduces a standard color chart as a reference, acquires a reference image containing the standard color chart, and calculates the component offset value of each color block under the current light source conditions. The white balance correction algorithm calculates the gain coefficients of the red, green, and blue channels of the image as a whole, so that the channel values ​​of the white area in the image tend to be consistent.

3. The method for image classification of traditional Chinese medicine materials according to claim 1, characterized in that, The knowledge graph construction process in step 2 includes: extracting medicinal material entities and their relationships from pharmacopoeias, traditional Chinese medicine classics and prescription databases. Each medicinal material node is associated with attribute information, which includes the name, aliases, properties, meridians, and efficacy categories of the medicinal material. The principal drug is defined as the highest-level node in the prescription. Under the principal drug node, downward edges are established according to the logic of the assistant drugs assisting the principal drug. Based on the logic of the adjuvant drugs cooperating with the principal and assistant drugs, and the logic of the guiding drug guiding the drug power or harmonizing the drugs, multi-level connections are established in sequence. Each edge in the knowledge graph is assigned a weight value, which is determined by the co-occurrence frequency of medicinal materials in historical prescriptions, the closeness of their compatibility, and the synergistic chemical reaction between the drugs, thus forming a structured knowledge system with reasoning capabilities.

4. The method for image classification of traditional Chinese medicine materials according to claim 1, characterized in that, The feature extraction and fusion process of the multi-scale feature fusion network in step 3 includes: using multiple parallel convolutional branches to process image patches with different receptive field sizes respectively; the first convolutional branch uses a 3×3-dimensional convolutional kernel to extract local texture features; the second convolutional branch uses a 5×5-dimensional convolutional kernel to extract medium-scale features; and the third convolutional branch uses a 7×7-dimensional convolutional kernel to capture overall morphological features. A channel attention mechanism is introduced into the convolutional network. Global average pooling is performed on the feature map of each channel to obtain global statistical information of the channel dimension. The importance weight of each channel is learned through two fully connected layers. The importance weight is applied to the original feature map to dynamically weight the features at each scale. The feature vectors extracted from each branch are concatenated along the channel dimension to generate a unified visual feature vector; By employing a feature pyramid connection structure, the abstract features output by the deep network are upsampled to ensure that their spatial resolution is consistent with that of the shallow feature map. The deep semantic features and shallow geometric features are combined by adding them pixel by pixel.

5. The method for image classification of traditional Chinese medicine materials according to claim 1, characterized in that, The cross-modal alignment process in step 4 includes: converting the image region into a visual vector in a continuous numerical space through a visual encoder, and converting the symbolic information in the knowledge graph into a semantic embedding vector of the same dimension through a semantic embedding module. A contrastive learning strategy is adopted to simultaneously optimize the parameters of the visual encoder and the semantic embedding module. For medicinal materials belonging to the same category, the Euclidean distance between their visual feature vectors and corresponding semantic embedding vectors in the joint space is minimized. For medicinal materials belonging to different categories, guide their vector spacing to tend to the maximum; By calculating the dot product of the visual vector and the semantic embedding vector and then normalizing it, a similarity score reflecting the degree of matching between the candidate region image and the medicinal material category is obtained.

6. The method for image classification of traditional Chinese medicine materials according to claim 1, characterized in that, The causal inference mechanism in step 5 is executed in the following way: the medicinal material stacking scene is modeled based on Bayesian network, and the co-occurrence frequency, compatibility, occlusion ratio and lighting conditions of medicinal materials are used as observable variables, and the real category state of medicinal materials is used as latent variables. When the visual features of a candidate region are not complete, the distribution of medicinal materials around the candidate region is analyzed and the compatibility rules in the knowledge graph are retrieved. The logical causal relationship between the identified medicinal materials and the region to be identified is calculated. The initial visual classification probability is used as the prior distribution. Combined with the causal correction probability generated based on the knowledge graph, the adjusted classification confidence is calculated through the posterior probability. The logical relationship between the medicinal materials is used to make up for the lack of visual information caused by physical occlusion.

7. The method for image classification of traditional Chinese medicine materials according to claim 1, characterized in that, The final medicinal material identification result output in step 6 also includes the following: through the node hierarchy relationship in the knowledge graph, the identified medicinal materials are automatically assigned role attributes in the current prescription scenario, indicating whether they are principal, assistant, adjuvant, or guiding herbs. The location information includes the center point pixel coordinates of each medicinal material target, the width and height of the bounding rectangle, and the pixel-level mask projection; The area ratio of medicinal materials in the image is calculated using the mask projection, and the weight ratio of each medicinal material is estimated by combining the density constant of the medicinal materials.

8. The method for image classification of traditional Chinese medicine materials according to claim 1, characterized in that, The method also includes a model optimization and update mechanism: when the highest confidence level of the system output is lower than the preset threshold and the identified combination of medicinal materials violates the contraindications of traditional Chinese medicine, the manual review interface is triggered, and the manually corrected labels along with the original image data are fed back to the training dataset to fine-tune the multi-scale feature fusion network and cross-modal alignment module. The knowledge graph supports dynamic updates. By monitoring pharmacopoeia updates or clinical research progress, nodes can be added or removed or edge weights can be modified in the knowledge graph. The semantic embedding space mapping can be updated to adapt the system to new TCM knowledge. The multi-scale feature fusion network is lightweighted and compressed. The standard convolution is replaced by depthwise separable convolution to decouple spatial convolution from channel convolution, and channel pruning technique is applied to remove redundant channels.

9. The method for image classification of traditional Chinese medicine materials according to claim 1, characterized in that, The method also involves feature enhancement for different application scenarios: in the case of medicinal material slice recognition, the knowledge graph adds processing status attributes, establishes a correlation chain between the medicinal material prototype and processing method and visual change features, and adds a color moment feature extraction branch to the multi-scale feature fusion network to capture the color shift caused by processing. In the context of warehousing and logistics identification, the knowledge graph introduces nodes for counterfeit and easily confused products, and establishes a mapping relationship between origin and visual features. The causal inference mechanism introduces consistency within the same batch as a causal variable, and increases the confidence of origin identification based on batch consistency logic. In the microscopic recognition scenario, high-resolution imaging equipment is coupled with an optical microscope to acquire microscopic images of medicinal powder. The knowledge graph contains microscopic structural features of calcium oxalate clusters, stone cells, and vessel types. The causal inference mechanism determines whether there is artificial doping by calculating the co-occurrence contradiction between heterogeneous microscopic features and main medicinal material features.

10. The method for image classification of traditional Chinese medicine materials according to claim 1, characterized in that, The method also involves production monitoring and environmental adaptive processing: In the scenario of material feeding monitoring, the knowledge graph adds material feeding ratio constraint rules, and the system accumulates the area or estimated weight of each identified medicinal material in real time. When the accumulated amount of a certain medicinal material reaches the upper limit required by the prescription, the causal inference mechanism lowers the recognition probability of the medicinal material in subsequent images and automatically compares the recognition results with the production execution system. In the scenario of identifying wild medicinal herbs, the collected images are preprocessed with adaptive histogram equalization to eliminate interference from natural light fluctuations. The knowledge graph is enhanced with geographic information system attributes, and the device's location information is used as an environmental causal variable. The identification weights of regional medicinal herb varieties are adjusted based on altitude and latitude information. In the teaching recognition scenario, low-resolution images are converted into high-resolution images through image super-resolution reconstruction algorithms, and the alignment results of visual features and semantic information are used to trigger augmented reality display, overlaying the position of the medicinal material in the monarch-minister-assistant-guide system at the physical location of the medicinal material.

Images