Whole-process Chinese herbal medicine intelligent category identification device and operation method

Abstract

The invention discloses a full-process Chinese herbal medicine intelligent category identification device and an operation method. The device comprises a mechanical arm, a conveyor belt, a sampling box, a hardware mainboard, a multifunctional expansion board, an HDMI external screen, a mainboard power switch, an LED annular lamp, a camera and a mobile power supply. The mechanical arm is used for intelligently grabbing the Chinese herbal medicine and grabbing the Chinese herbal medicine from the placing frame to the sampling plate; the conveying belt is used for automatically conveying Chinese herbal medicines, and the Chinese herbal medicines are conveyed into the sampling box through the conveying sampling plate; the sampling box is used for providing a Chinese herbal medicine shooting environment and is internally provided with a plurality of parts to form a complete image acquisition device; the hardware mainboard is used for controlling other parts to work cooperatively and is internally provided with multiple intelligent algorithms for intelligent processing of Chinese herbal medicine images. The intelligent Chinese herbal medicine picking and transporting function, the intelligent Chinese herbal medicine image collecting function and the intelligent Chinese herbal medicine category identifying function are achieved, the Chinese herbal medicine images can be efficiently collected, and the Chinese herbal medicine categories can be intelligently identified.

Description

Technical Field

[0001] This invention belongs to the field of image processing and classification, and relates to intelligent grasping and transportation, image acquisition and processing and image classification technology of Chinese herbal medicines. Specifically, it relates to an intelligent classification identification device and operation method for Chinese herbal medicines throughout the entire process. Background Technology

[0002] Identification of Chinese herbal medicines plays a crucial role in determining their efficacy and ensuring their correct use. Current traditional methods for identifying Chinese herbal medicines include morphological identification, microscopic identification, and physicochemical identification. These methods largely rely on manual techniques, are cumbersome and complex, and suffer from low efficiency, high cost, and strong subjectivity. Furthermore, they are difficult to use for identifying large quantities of Chinese herbal medicines. Therefore, designing a method for rapidly identifying large quantities of Chinese herbal medicines is both significant and challenging.

[0003] In recent years, the continuous development of artificial intelligence (AI) technology has enabled efficient and accurate plant classification through intelligent analysis and feature extraction of plant images using computer vision techniques. This advancement provides an innovative solution for the rapid identification of traditional Chinese medicine (TCM) herbs. By combining AI technology with TCM herb identification and leveraging computer vision techniques such as image classification, the identification of TCM herbs can be made intelligent and efficient. Currently, deep learning methods have achieved significant results in computer vision tasks such as image classification. Analyzing and learning images using deep learning methods can train highly efficient intelligent models. However, TCM herb identification models trained using deep learning methods require computer hardware and are difficult to directly apply to real-world TCM herb scenarios. Summary of the Invention

[0004] Purpose of the invention: In order to overcome the shortcomings of the existing technology, this invention provides a fully intelligent classification identification device and operation method for Chinese herbal medicines, which realizes the functions of automatic grasping and transportation of Chinese herbal medicines, intelligent acquisition and processing of Chinese herbal medicine images, and intelligent classification of Chinese herbal medicine images. It can quickly and efficiently identify the categories of Chinese herbal medicines and apply deep learning models to real-world scenarios.

[0005] Technical solution: To achieve the above objectives, the present invention provides an intelligent classification identification device for Chinese herbal medicines throughout the entire process, including a robotic arm, a conveyor belt, a sampling box, a hardware motherboard, a multi-functional expansion board, an HDMI external screen, a motherboard power switch, an LED ring light, a camera, and a power bank.

[0006] The robotic arm is used to intelligently grasp Chinese herbal medicines, picking them from the placement box and placing them on the sampling plate;

[0007] The conveyor belt is used for automatic transport of Chinese herbal medicines, and the Chinese herbal medicines are transported into the sampling box through the transport sampling plate;

[0008] The sampling box is used to provide an environment for photographing Chinese herbal medicines;

[0009] The hardware motherboard is used to control the coordinated work of other components and has built-in multiple intelligent algorithms for intelligent processing of Chinese herbal medicine images.

[0010] The multi-functional expansion board is used to expand the motherboard interface, enhance the functionality of the motherboard, and enable the motherboard to connect and control more components.

[0011] The HDMI external screen is used to display the device control software, which facilitates device function control and displays the device's operating stages and results.

[0012] The motherboard power switch is used to control the motherboard to turn on and off;

[0013] The LED ring light is used to provide a stable light source for the shooting environment;

[0014] The camera is used to capture images of Chinese herbal medicine samples;

[0015] The portable power supply provides power to the internal components of the device, ensuring that the device can work for a long time in actual application scenarios and guaranteeing the device's battery life.

[0016] Furthermore, the robotic arm is connected to the hardware motherboard via a connecting cable. The movement and gripping / releasing actions of the robotic arm are controlled by servo motors. The servo motors of the robotic arm are controlled by the hardware motherboard, and the main body of the robotic arm has its own lithium battery box, which powers the hardware motherboard through the lithium battery.

[0017] Furthermore, the conveyor belt consists of a DC geared motor, an industrial PVC conveyor belt, and an aluminum alloy structural support. The motor wires are connected to the hardware motherboard, so that the operation of the motor is controlled by the motherboard. The conveyor belt's transport direction and speed can be controlled through the motherboard.

[0018] Furthermore, the acquisition box is a semi-enclosed box, and the inside of the acquisition box is equipped with LED lights, a camera, a hardware motherboard, and a power bank, which together form a good sample image shooting environment and can be used independently as a Chinese herbal medicine image acquisition device.

[0019] Furthermore, the hardware motherboard is located at the bottom of the sampling box. The hardware motherboard includes a first motherboard and a second motherboard. The first motherboard is responsible for controlling the robotic arm, conveyor belt, and camera to realize the intelligent grasping function of Chinese herbal medicine, the intelligent transportation function of Chinese herbal medicine, and the image acquisition function of Chinese herbal medicine samples. At the same time, the first motherboard transmits the acquired image data to the background database. After receiving the acquisition completion signal, the second motherboard retrieves the corresponding acquired image from the background database and performs processing on the acquired image, such as Chinese herbal medicine target detection, single Chinese herbal medicine image cropping, image quality screening, and intelligent classification of Chinese herbal medicine images. The process data generated during the processing is synchronously uploaded to the background database.

[0020] Furthermore, the multi-functional expansion board is installed on the first motherboard, expanding the motherboard's external interfaces and providing servo cable interfaces and motor interfaces, facilitating the motherboard's connection to and control of the robotic arm and conveyor belt.

[0021] Furthermore, the external HDMI screen is connected to the USB port of the first motherboard via a micro-USB cable, powered by the motherboard, and enables touch screen functionality. It is also connected to the HDMI output port of the first motherboard via an HDMI cable to display the control software interface.

[0022] Furthermore, the LED ring light, arranged on the top of the sampling box, provides 5600K white light with a power of 5-10W, supporting long-term operation.

[0023] Furthermore, the camera is a high-definition camera used to capture images of Chinese herbal medicine samples. It is positioned on the top of the sampling box and connected to the hardware motherboard via a USB cable, and is controlled by the hardware motherboard.

[0024] Furthermore, the mobile power supply is located at the bottom of the sampling box, has a capacity of 20,000mAh, and provides a maximum power of 30W to meet the working requirements of the hardware motherboard and LED lights.

[0025] This invention also provides a method for operating a fully automated intelligent classification and identification device for traditional Chinese medicine, comprising the following steps:

[0026] S1: Turn on the motherboard power switch. The control software login interface is displayed on the HDMI screen. The user logs in by entering the user ID and password. After confirming the user information, the user enters the function control interface.

[0027] S2: Select the real-time monitoring function to display the real-time situation inside the acquisition box on the HDMI screen;

[0028] S3: Select the grasping and transport function. The first motherboard controls the robotic arm to grasp the Chinese herbal medicine from the herbal medicine placement box. Then the robotic arm places the Chinese herbal medicine on the sampling plate. Next, the conveyor belt transports the sampling plate to the collection interior. After the user observes the Chinese herbal medicine sample on the sampling plate, select the shooting and collection function. The first motherboard controls the camera to collect images of the sample and saves the image results. Simultaneously, the image data is transmitted to the database in the background and a collection completion signal is sent to the second motherboard.

[0029] S4: After receiving the acquisition completion signal, the second motherboard retrieves the corresponding acquired image from the database, then uses the Chinese herbal medicine target detection model to perform target detection on the image, obtains the location information of a single Chinese herbal medicine, and then uses the region cropping algorithm to crop the image of a single Chinese herbal medicine from the image, and saves the detection result and the cropping result to the database simultaneously.

[0030] S5: After the second motherboard completes the detection and capture task, it performs quality evaluation and screening on all single Chinese herbal medicine images. It scores the images based on the angle, incompleteness, and other conditions of the Chinese herbal medicines in the images. Based on the scoring results, the high-scoring images are input into the Chinese herbal medicine image classifier for Chinese herbal medicine category determination. During this process, the image scoring results and the Chinese herbal medicine category determination results are saved to the background database in real time.

[0031] S6: During steps S4 to S5, the user can control the second motherboard to perform target detection and capture, quality evaluation and screening, and identification of Chinese herbal medicine categories step by step. The image processing results of each step will be displayed on the HDMI screen for the user to observe.

[0032] Further, the specific method for target detection and cropping of the image in step S4 is as follows: A Chinese herbal medicine target detection model is trained using a YOLO v5s network and image patch data of medicinal herbs tagged with medicinal herbs. This model is then used to detect medicinal herbs in images containing them, determining the region information of individual medicinal herbs in the image, including the center point coordinates (x, y). c ,y c ) and the width w of the area c and height h c The detection formula for the Chinese herbal medicine target detection model is as follows:

[0033]

[0034] To ensure the integrity of individual Chinese herbal medicines in the subsequently extracted images, the regions of medicinal materials detected by the Chinese herbal medicine target detection model were observed. Through multiple experiments, an expansion coefficient k was designed to expand the regions. The specific coordinates (x, y, k) of the upper left and lower right corners of the expanded regions were then determined. left ,y left ) and (x right ,yright The image is extracted from the original image using two coordinates.

[0035] The specific coordinates (x, y) of the top left and bottom right corners of the expanded region left ,y left ) and (x right ,y right The formula for calculating ) is:

[0036]

[0037] Further, the specific method for quality evaluation and screening of images in step S5 is as follows: A trained angle quality evaluation model and a missing quality evaluation model are used to score the quality of a single herbal medicine image, obtaining quality scores s1 and s2, representing the quality scores of the herbal medicine in the image based on angle and missing quality criteria, respectively. Then, the two scores are combined to obtain a comprehensive quality score s. Next, the comprehensive quality score s is compared with a set threshold m. If it is greater than the threshold, the image quality meets the standard; otherwise, the image quality is unqualified. The specific formula for the quality evaluation and screening process is as follows:

[0038] s = s1 + s2

[0039]

[0040] Furthermore, the specific method for identifying the Chinese herbal medicine category in step S5 is as follows: First, a Chinese herbal medicine type discrimination model is used to make a preliminary type determination of the Chinese herbal medicine in the image. Then, according to the type, the corresponding Chinese herbal medicine classification method is selected, and the selected method is used to perform fine-grained classification of the Chinese herbal medicine in the image.

[0041] The device in this invention can automatically grasp and transport Chinese herbal medicine samples, simplifying the manual sorting process. Simultaneously, the device can automatically acquire images of the Chinese herbal medicine samples and use target detection technology to obtain images of individual herbs. Through image quality evaluation models and Chinese herbal medicine type determination models, quality and type labels are assigned to the images, ensuring that high-quality images of the corresponding type are provided for subsequent refined identification of Chinese herbal medicines. The relevant deep learning models are all integrated into the device's hardware motherboard, and the device itself is highly mobile, enabling deep learning technology to be better applied in practical scenarios related to Chinese herbal medicines.

[0042] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0043] 1. This invention can control the robotic arm and conveyor belt through a hardware motherboard to realize the automatic grasping and transportation of Chinese herbal medicines. The hardware motherboard can also control the camera to collect images of Chinese herbal medicine samples. After the collection is completed, the hardware motherboard will call the internally embedded intelligent algorithm to intelligently process the images and realize intelligent identification of Chinese herbal medicine categories.

[0044] 2. This invention utilizes a Chinese herbal medicine target detection model to detect targets in images. Then, it uses a region cropping algorithm to extract images of individual Chinese herbal medicines. Next, it evaluates the image quality of each individual Chinese herbal medicine image and retains high-quality images based on their quality scores for subsequent identification of Chinese herbal medicine categories, thereby achieving intelligent acquisition and screening of high-quality Chinese herbal medicine images.

[0045] 3. This invention utilizes a Chinese herbal medicine image classification method for intelligent identification of Chinese herbal medicine categories. It uses corresponding classification methods for different types of Chinese herbal medicine, thereby improving the efficiency and accuracy of Chinese herbal medicine category identification.

[0046] 4. Existing deep learning-based identification models require the use of portable hardware devices such as computers and servers. Furthermore, image acquisition is necessary before using the model, and in conventional use, image acquisition and model identification are two separate processes. The identification device designed in this invention integrates these two processes. It utilizes a robotic arm and conveyor belt for intelligent grasping and transport of medicinal materials, simplifying the manual handling and placement of herbs during device use. The acquisition box provides a stable shooting environment for effective image acquisition of medicinal materials. The hardware motherboard provides the operating environment for relevant intelligent algorithms and models, enabling the device's specific functions, including intelligent collaboration between hardware devices, intelligent image processing, and intelligent identification of medicinal materials. The device designed in this invention is disassembled and reassembled, facilitating use in various practical scenarios and solving the problem that existing Chinese herbal medicine identification models are difficult to directly apply to real-world applications of Chinese herbal medicine. Attached Figure Description

[0047] Figure 1 This is the hardware design diagram of the intelligent classification identification device for Chinese herbal medicines throughout the entire process of this invention;

[0048] Figure 2 This is an algorithm flowchart of the intelligent classification identification device for traditional Chinese medicines throughout the entire process of this invention;

[0049] Figure 3 This is a physical image of the intelligent herbal medicine classification identification device for the entire process of this invention;

[0050] Figure 4 This is a diagram of the mainboard components of the intelligent classification and identification device for traditional Chinese medicine throughout the entire process of this invention;

[0051] Figure 5This is a flowchart of the control software operation of the intelligent classification identification device for Chinese herbal medicines throughout the entire process of this invention;

[0052] Figure 6 This is a schematic diagram of the structure of an image quality assessment network. Detailed Implementation

[0053] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0054] like Figure 1 As shown, the present invention provides an intelligent classification identification device for Chinese herbal medicines throughout the entire process, including a first motherboard 1, a second motherboard 2, a robotic arm 3, a conveyor belt 4, a mobile power supply 5, an LED ring light 6, a camera 7, a sampling box 8, and a server 9.

[0055] The robotic arm 3 mainly relies on servo motors for movement and grasping, including metal tooth micro servo motors, high torque servo motors and 15KG high torque servo motors. Each servo motor is connected to the first motherboard 1 through a connecting cable, so that the robotic arm 3 is controlled by the first motherboard 1 and can perform intelligent grasping.

[0056] The movement of the conveyor belt 4 is controlled by a DC geared motor. The motor is connected to the first main board 1 through a motor wire. The first main board 1 can control the running direction and speed of the conveyor belt 4. The conveyor belt 4 is 71 cm long, 12 cm wide, and 6.7 cm high. The overall net weight is about 3.7 kg. The working voltage is 12V, and the acceptable operating load is about 9 kg.

[0057] The sampling box 8 is a semi-enclosed box. The first main board 1, the second main board 2, the power bank 5, the LED ring light 6, and the camera 7 are all installed inside the sampling box 8, forming a relatively good sample shooting and collection environment. The box is about 40 cm long, about 40 cm wide, and about 30 cm high.

[0058] The first motherboard 1 and the second motherboard 2 can communicate with each other and transmit simple control signals. The first motherboard 1 can control the robotic arm 3, conveyor belt 4, and camera 7 to achieve corresponding functions. Users can select corresponding functions through the touch screen, allowing the first motherboard 1 to control the corresponding components to complete the corresponding functions. The second motherboard 2 is embedded with intelligent algorithms related to Chinese herbal medicine target detection, Chinese herbal medicine image quality evaluation, and Chinese herbal medicine category identification. It is responsible for processing the collected images of Chinese herbal medicine samples. The invocation of intelligent algorithms in the second motherboard 2 is controlled by the signals transmitted by the first motherboard 1. Users can control the image processing process of the second motherboard 2 by controlling the first motherboard 1. Both the first motherboard 1 and the second motherboard 2 can transmit and receive data to the background database. Image data is transmitted between the two motherboards through the database, ensuring data storage and archiving while ensuring the consistency and integrity of image data between the two motherboards.

[0059] The external HDMI screen is 7 inches in size with a resolution of 1024*600 pixels. It is connected to the USB port of the first motherboard 1 via a micro-USB cable. The first motherboard 1 is responsible for powering the screen and enabling the touch screen function. It is also connected to the HDMI output port of the first motherboard 1 via an HDMI cable to display the device control software interface.

[0060] The LED ring light 6 is arranged on the top of the sampling box 8. The LED ring light 6 has a working power of 5-10W, supports USB interface power supply, provides white light, and has a color temperature of 5600K, providing a stable light source for sample imaging and collection, and improving the collection effect.

[0061] Camera 7 is a high-definition camera, which is connected to the USB interface of the first motherboard 1 via a USB cable and is controlled by the first motherboard 1. Camera 7 has a resolution of 640*480 pixels and a bitstream type of YUY2. It is used to capture images of Chinese herbal medicine samples.

[0062] The power bank 5 provides stable power to the hardware motherboard, has a 20000mAh capacity, and provides a maximum power of 30W, allowing the hardware motherboard to work continuously for a long time and ensuring the device's battery life in real-world application scenarios.

[0063] like Figure 2 As shown, the device first uses a traditional Chinese medicine (TCM) target detection model to detect targets in the captured images of TCM sample to determine the specific location of each individual herb. Then, a region cropping algorithm is used to dilate the detected region to ensure the integrity of the sample in the image, followed by cropping individual herb images from the image. Next, a TCM image quality assessment algorithm is used to screen the individual herb images, retaining high-quality images. Finally, a TCM category identification algorithm is used to identify the category of the high-quality herb images, determining the specific category information of the TCM sample.

[0064] like Figure 3 As shown, the various components of the device and their actual layout are illustrated. The key parts of the entire device are the robotic arm, the conveyor belt, and the collection box. The robotic arm is positioned next to the conveyor belt, with a herbal medicine placement frame on its left for placing herbal medicine samples. The robotic arm's workflow is as follows: upon receiving a grasping signal, it moves its end effector to the herbal medicine placement frame, grasps the herbal medicine sample, then moves the end effector above the sampling plate, places the grasped sample on the sampling plate, and finally sends a grasping completion signal to the main board. The conveyor belt is placed in the middle of the sampling box, extending outside the box, and the sampling plate is mounted on the conveyor belt to compensate for the insufficient width of the conveyor belt. The conveyor belt's workflow is as follows: upon receiving the grasping completion signal, the main board controls the conveyor belt to transport the sampling plate into the collection box, and finally, after the collection box completes the sample image acquisition and processing task, it transports the sampling plate out. The collection box contains a camera, LED lights, a hardware main board, and a power bank, enabling the acquisition and processing of herbal medicine sample images and the identification of herbal medicine categories.

[0065] like Figure 4 As shown, both the first and second motherboards, along with the multi-functional expansion board, provide abundant interfaces, enabling connection to multiple different components and allowing the motherboard to control multiple hardware components to work collaboratively. The first motherboard features a quad-core 64-bit CPU (ARM Cortex-A72), 8GB of RAM, and is equipped with a multi-functional expansion board that expands its interfaces, providing multiple motor and servo interfaces for easy connection to robotic arms and conveyors. The second motherboard is primarily used for image processing; its interfaces are not currently connected to any components, leaving ample space for future expansion. The second motherboard features a quad-core ARM Cortex-A57 CPU and a 128-core NVIDIA Maxwell architecture GPU supporting CUDA acceleration, enabling the execution of intelligent processing algorithms.

[0066] like Figure 1 and Figure 5 As shown, this embodiment uses the above-mentioned intelligent classification device for the entire process of traditional Chinese medicine (TCM) for the intelligent acquisition and classification of TCM images, and provides a method for operating the intelligent acquisition and classification device for TCM images, including the following steps:

[0067] S1: Turn on the motherboard power switch. The control software login interface is displayed on the HDMI screen of server 9. The user logs in by entering the user ID and password. After confirming the user information, the user enters the function control interface. The robotic arm 3, conveyor belt 4, LED ring light 6, and camera 7 all enter the working state.

[0068] S2: Select the real-time monitoring function. The first motherboard 1 controls the camera 7 to monitor the inside of the acquisition box 8 in real time, and the server 9 displays the real-time situation inside the acquisition box on its HDMI screen.

[0069] S3: Select the grasping and transport function. The first motherboard 1 controls the robotic arm 3 to grasp the Chinese herbal medicine from the herbal medicine placement box, and then the robotic arm 3 places the Chinese herbal medicine on the sampling plate. Then the conveyor belt 4 transports the sampling plate to the collection chamber. After the user observes the Chinese herbal medicine sample on the sampling plate, select the shooting and collection function. The first motherboard 1 controls the camera 7 to capture images of the sample, save the image results, and simultaneously transmit the image data to the background database, and send a collection completion signal to the second motherboard 2.

[0070] S4: After receiving the acquisition completion signal, the second motherboard 2 retrieves the corresponding acquired image from the database, then uses the Chinese herbal medicine target detection model to perform target detection on the image, obtains the location information of a single Chinese herbal medicine, and then uses the region cropping algorithm to crop the image of a single Chinese herbal medicine from the image, and saves the detection result and the cropping result to the database simultaneously.

[0071] The specific method for Chinese herbal medicine target detection and region extraction is as follows: A Chinese herbal medicine target detection model is trained using a YOLO v5s network and image patch data of herbs tagged with herbs. This model is then used to detect herbs in images containing them, determining the region information of individual herbs in the image, including the center point coordinates (x, y, y). c ,y c ) and the width w of the area c and height h c The detection formula for the Chinese herbal medicine target detection model is as follows:

[0072]

[0073] To ensure the integrity of individual Chinese herbal medicines in the subsequently extracted images, and observing the medicinal herb regions detected by the herbal medicine target detection model, a dilation coefficient k was designed through multiple experiments to dilate the regions. The specific coordinates (x, y, k) of the upper left and lower right corners of the dilated region are shown below. left ,y left ) and (x right ,y right The calculation formula is shown below. Using two coordinates, the image can be extracted from the original image.

[0074]

[0075] S5: After the second motherboard 2 completes the detection and capture task, it evaluates the quality of all single Chinese herbal medicine images. It scores the images based on the angle, defects, and other conditions of the Chinese herbal medicines in the images. Based on the scoring results, the high-scoring images are input into the Chinese herbal medicine image classifier for Chinese herbal medicine category determination. During this process, the image scoring results and the Chinese herbal medicine category determination results are saved to the background database in real time.

[0076] The method for quality assessment and screening of images of traditional Chinese medicine is as follows: An image quality assessment network based on deep color feature fusion was designed for traditional Chinese medicine. The overall network structure is as follows: Figure 6 As shown, the image quality assessment network consists of three parts in its composition and operation:

[0077] 1) NET1: Extracting S color channel features f from HSV and Lab color spaces S L color channel features f L and b color channel features f b Then, using the Local Binary Patterns (LBP) feature matrix f calculated from the grayscale image... LBP The three features are preprocessed separately, and then the three preprocessed features are fused to obtain feature f. SLb Then, convolutional networks are used to further mine the quality-related semantic features;

[0078] 2) NET2: Extract features f from the R, G, and B channels of the original image's RGB color space. R f G f B Feature fusion is performed, and then the network is used to further analyze the quality-related semantic features in the original image;

[0079] 3) Feature fusion and quality assessment: The quality-related features extracted from NET1 and NET2 are fused to achieve a comprehensive description of image quality.

[0080] Using the aforementioned network structure, angle quality assessment models and missing quality assessment models are trained on a dataset of Chinese herbal medicines with angle distortion and a dataset of Chinese herbal medicines with missing characteristics, respectively. These two models can respectively score the quality of individual Chinese herbal medicine images, obtaining quality scores s1 and s2, representing the quality scores of the Chinese herbal medicine in the image based on angle and missingness assessment criteria, respectively. Then, the two scores are combined to obtain a comprehensive quality score s. Next, the comprehensive quality score s is compared with a set threshold m. If it is greater than the threshold, the image quality meets the standard; otherwise, the image quality is unacceptable. The specific formula for the quality assessment and screening process is as follows:

[0081]

[0082] The specific method for identifying images of traditional Chinese medicine (TCM) herbs is as follows: A TCM herb type discrimination model is trained on a dataset containing images of different types of TCM herbs using a conventional network. This model can analyze the input TCM herb images and determine the type of TCM herb to which the image belongs. The model's determination formula is shown below:

[0083] Model C (image) = kind

[0084] Different deep learning networks are designed for different types of Chinese herbal medicines and trained on corresponding Chinese herbal medicine image datasets to obtain corresponding Chinese herbal medicine category identification models. The corresponding Chinese herbal medicine category identification models can accurately analyze images of fixed types of Chinese herbal medicines to identify specific information about the Chinese herbal medicines, including details such as variety, origin, and grade. The identification formula of the Chinese herbal medicine category identification model is as follows:

[0085]

[0086] The specific process of the device in identifying images of traditional Chinese medicine (TCM) herbs is as follows: First, a TCM herb type discrimination model is used to make a preliminary type determination of the TCM herbs in the image. Then, according to the type, a corresponding TCM herb category identification model is selected, and the selected model is used to perform fine-grained identification of the TCM herbs in the image. Currently, the device is equipped with TCM herb chrysanthemum identification model, TCM herb dried tangerine peel identification model, and TCM herb slice identification model. Among them, the TCM herb slice identification model mainly targets four types of medicinal material slices: Bupleurum, Pinellia ternata, Poria cocos, and Codonopsis pilosula. The accuracy of each classification method on the corresponding dataset is 0.9555, 0.9056, and 0.9503, respectively.

[0087] S6: During steps S4 to S5, the user can control the second motherboard to perform target detection and capture, quality evaluation and screening, and identification of Chinese herbal medicine categories in two steps. The image processing results of each step will be displayed on the HDMI screen for the user to observe.

Claims

1. A fully automated intelligent classification and identification device for traditional Chinese medicine, characterized in that, Includes robotic arm, conveyor belt, sampling box, hardware motherboard, multi-functional expansion board, HDMI external screen, motherboard power switch, LED ring light, camera, and power bank; The robotic arm is used to intelligently grasp Chinese herbal medicines, picking them from the placement box and placing them on the sampling plate; The conveyor belt is used for automatic transport of Chinese herbal medicines, and the Chinese herbal medicines are transported into the sampling box through the transport sampling plate; The sampling box is used to provide an environment for photographing Chinese herbal medicines; The hardware motherboard is used to control the coordinated work of other components and has built-in multiple intelligent algorithms for intelligent processing of Chinese herbal medicine images. The multi-functional expansion board is used to expand the motherboard interface, enhance the functionality of the motherboard, and enable the motherboard to connect and control more components. The HDMI external screen is used to display the device control software, which facilitates device function control and displays the device's operating stages and results. The motherboard power switch is used to control the motherboard to turn on and off; The LED ring light is used to provide a stable light source for the shooting environment; The camera is used to capture images of Chinese herbal medicine samples; The portable power supply provides power to the internal components of the device, ensuring that the device can work for a long time in actual application scenarios and guaranteeing the device's battery life.

2. The intelligent classification identification device for traditional Chinese medicines throughout the entire process according to claim 1, characterized in that, The robotic arm is connected to the hardware motherboard via a connecting cable. The movement and gripping actions of the robotic arm are controlled by servo motors. The servo motors of the robotic arm are controlled by the hardware motherboard, and the main body of the robotic arm has its own lithium battery box, which powers the hardware motherboard through the lithium battery.

3. The intelligent classification identification device for traditional Chinese medicines throughout the entire process according to claim 1, characterized in that, The acquisition box is a semi-enclosed box. Inside the acquisition box are installed LED lights, cameras, hardware motherboards, and power banks, which together form a good sample image shooting environment and can be used independently as a Chinese herbal medicine image acquisition device.

4. The intelligent classification identification device for traditional Chinese medicines throughout the entire process according to claim 3, characterized in that, The hardware motherboard is located at the bottom of the sampling box. The hardware motherboard includes a first motherboard and a second motherboard. The first motherboard is responsible for controlling the robotic arm, conveyor belt, and camera to realize the functions of intelligent grasping, intelligent transportation, and image acquisition of Chinese herbal medicines. At the same time, the first motherboard transmits the acquired image data to the background database. After receiving the acquisition completion signal, the second motherboard retrieves the corresponding acquired image from the background database, processes the acquired image, and the process data generated during the processing is synchronously uploaded to the background database.

5. The intelligent classification identification device for traditional Chinese medicines throughout the entire process according to claim 4, characterized in that, The multi-functional expansion board is installed on the first motherboard, expanding the motherboard's external interfaces and providing servo cable interfaces and motor interfaces, facilitating the motherboard's connection and control of the robotic arm and conveyor belt.

6. The operating method of the intelligent classification identification device for traditional Chinese medicine throughout the entire process according to claim 1, characterized in that, Includes the following steps: S1: Turn on the motherboard power switch. The control software login interface is displayed on the HDMI screen. The user logs in by entering the user ID and password. After confirming the user information, the user enters the function control interface. S2: Select the real-time monitoring function to display the real-time situation inside the acquisition box on the HDMI screen. S3: Select the grabbing and transporting function. The first motherboard controls the robotic arm to grab Chinese herbs from the Chinese herbal medicine placement box. Then the robotic arm places the Chinese herbal medicine on the sampling plate. Next, the conveyor belt transports the sampling plate to the inside of the collection box. After the user observes the Chinese herbal medicine sample on the sampling plate, select the shooting and collection function. The first motherboard controls the camera to collect images of the sample and saves the image results. The image data is simultaneously transmitted to the database in the background and a collection completion signal is sent to the second motherboard. S4: After receiving the acquisition completion signal, the second motherboard retrieves the corresponding acquired image from the database, then uses the Chinese herbal medicine target detection model to perform target detection on the image, obtains the location information of a single Chinese herbal medicine, and then uses the region cropping algorithm to crop the image of a single Chinese herbal medicine from the image, and saves the detection result and the cropping result to the database simultaneously. S5: After the second motherboard completes the detection and capture task, it performs quality evaluation and screening on all single Chinese herbal medicine images, scores the images based on the information of the Chinese herbal medicine in the images, and inputs the high-scoring images into the Chinese herbal medicine image classifier for Chinese herbal medicine category determination based on the scoring results. During this process, the image scoring results and the Chinese herbal medicine category determination results are saved to the background database in real time. S6: During steps S4 to S5, the user controls the second motherboard to perform target detection and capture, quality evaluation and screening, and identification of Chinese herbal medicine categories step by step. The image processing results of each step will be displayed on the HDMI screen for the user to observe.

7. The operating method of the intelligent image acquisition and classification device for traditional Chinese medicine according to claim 6, characterized in that, The specific method for target detection and cropping in step S4 is as follows: A Chinese herbal medicine target detection model is trained using a YOLO v5s network and image patch data of medicinal herbs tagged with medicinal herbs. This model is then used to detect medicinal herbs in images containing them, determining the region information of individual medicinal herbs in the image, including the center point coordinates (x, y). c ,y c ) and the width w of the area c and height h c The detection formula for the Chinese herbal medicine target detection model is as follows: To ensure the integrity of individual Chinese herbal medicines in the subsequently extracted images, the regions of medicinal materials detected by the Chinese herbal medicine target detection model were observed. Through multiple experiments, an expansion coefficient k was designed to expand the regions. The specific coordinates (x, y, k) of the upper left and lower right corners of the expanded regions were then determined. left ,y left ) and (x right ,y right The image is extracted from the original image using two coordinates.

8. The operating method of the intelligent image acquisition and classification device for traditional Chinese medicine according to claim 7, characterized in that, The specific coordinates (x, y) of the upper left and lower right corners of the expanded region in step S4 are as follows: left ,y left ) and (x right ,y right The formula for calculating ) is:

9. The operating method of the intelligent image acquisition and classification device for traditional Chinese medicine as described in claim 6, characterized in that, The specific method for image quality evaluation and screening in step S5 is as follows: An image quality evaluation network based on deep color feature fusion is designed for traditional Chinese medicine, and it consists of three parts: 1) NET1: Extracting S color channel features f from HSV and Lab color spaces S L color channel features f L and b color channel features f b Then, using the local binary pattern feature matrix f calculated from the grayscale image... LBP The three features are preprocessed separately, and then the three preprocessed features are fused to obtain feature f. SLb Then, convolutional networks are used to further mine the quality-related semantic features; 2) NET2: Extract features f from the R, G, and B channels of the original image's RGB color space. R f G f B Feature fusion is performed, and then the network is used to further analyze the quality-related semantic features in the original image; 3) Feature fusion and quality assessment: The quality-related features extracted from NET1 and NET2 are fused to achieve a comprehensive description of image quality; Using the aforementioned network structure, angle quality assessment models and missing quality assessment models were trained on a dataset of Chinese herbal medicines with angle distortion and a dataset of Chinese herbal medicines with missing defects, respectively. These two models can respectively score the quality of individual Chinese herbal medicine images, obtaining quality scores s1 and s2, representing the quality scores of the Chinese herbal medicine in the image based on angle and missingness assessment criteria, respectively. Then, the two scores are combined to obtain a comprehensive quality score s. Next, the comprehensive quality score s is compared with a set threshold m. If it is greater than the threshold, the image quality meets the standard; otherwise, the image quality is unqualified. The specific formula for the quality assessment and selection process is as follows: s = s1 + s2 10. The operating method of the intelligent image acquisition and classification device for traditional Chinese medicine according to claim 6, characterized in that, The specific method for identifying the Chinese herbal medicine category in step S5 is as follows: First, a Chinese herbal medicine type discrimination model is used to make a preliminary type determination of the Chinese herbal medicine in the image. Then, according to the type, the corresponding Chinese herbal medicine classification method is selected, and the selected method is used to perform fine-grained classification of the Chinese herbal medicine in the image.

Images