Traditional Chinese medicine decoction piece identification method and interaction system based on image identification and natural language

Abstract

The invention relates to the field of data processing, and discloses a traditional Chinese medicine decoction piece identification method and interaction system based on image identification and natural language, and the method comprises the steps: an identification task management module obtains the type of an identification instruction according to the received identification instruction of a client, and carries out the identification of the identification instruction according to the identification information in the identification instruction; and the identification module generates an identification container corresponding to the identification information, the identification container corresponding to the identification information matches the identification result information corresponding to the identification information in the cloud data server, and the identification result information corresponding to the identification information is returned to the client to complete the identification of the traditional Chinese medicine decoction piece information. The traditional Chinese medicine decoction pieces can be identified more accurately, and identification errors are reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method and interactive system for identifying Chinese herbal medicine slices based on image recognition and natural language. Background Art

[0002] In the field of Traditional Chinese Medicine (TCM), the accurate identification of TCM slices, which are processed raw materials that can be directly used in TCM clinical prescriptions, is crucial for ensuring drug efficacy and safety. Traditionally, the identification of TCM slices relies primarily on the pharmacist's experience and visual inspection. While this method can guarantee accuracy to a certain extent, it is limited by the pharmacist's professional level, accumulated experience, and personal subjective judgment, and carries the risk of identification errors. Especially with the increasing variety of TCM materials and the increasing complexity of processing techniques, manual identification alone can no longer meet the needs of modern TCM development.

[0003] In recent years, with the rapid development of information technology, image recognition and natural language processing technologies have been gradually applied to the identification of Chinese herbal medicines, providing new approaches for the automated and intelligent recognition of Chinese herbal medicines. However, existing Chinese herbal medicine recognition systems often focus on either image recognition or text processing, lacking comprehensive recognition methods that effectively combine the two. In practical applications, users may provide both images and text information (such as the name) of a Chinese herbal medicine, but existing systems often only process one of these, limiting the accuracy and reliability of the recognition results.

[0004] Furthermore, the names of Chinese herbal medicines, including their official names and aliases, present challenges for natural language processing. Therefore, effectively integrating image recognition and natural language processing technologies to achieve efficient and accurate identification of Chinese herbal medicines has become a pressing issue. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method for identifying Chinese herbal medicine pieces based on image recognition and natural language, comprising the following steps: Step 1: The recognition task management module obtains the recognition instruction type based on the recognition instruction received from the client. If it is a single recognition instruction, it proceeds to step 2; if it is a mixed recognition instruction, it proceeds to step 3; In step 2, based on the identification information in the identification instruction, the identification module generates an identification container corresponding to the identification information. The identification container corresponding to the identification information matches the identification result information corresponding to the identification information on the cloud data server based on the identification information, and returns the identification result information corresponding to the identification information to the client, and then proceeds to step 6. Step 3: The recognition module generates an image processing container and a natural language processing container, which are connected to the client respectively. The image processing container obtains the image features of the Chinese herbal medicine slices according to the image of the Chinese herbal medicine slices in the recognition instruction sent by the client, and then proceeds to step 4. At the same time, the natural language processing container parses the text information of the Chinese herbal medicine slices in the recognition instruction sent by the client to obtain the text features of the Chinese herbal medicine slices, and then proceeds to step 4. Step 4: According to the obtained text features of the Chinese herbal medicine pieces, a standard image of the Chinese herbal medicine pieces corresponding to the text features of the Chinese herbal medicine pieces is matched on the cloud data server, the standard image features of the standard image of the Chinese herbal medicine pieces are obtained, and the image similarity with the image features of the image of the Chinese herbal medicine pieces in the recognition instruction is obtained. If the image similarity is greater than the set similarity threshold, the text information of the Chinese herbal medicine pieces in the recognition instruction matches the image of the Chinese herbal medicine pieces in the recognition instruction, and the corresponding recognition result information is returned to the client, and step 6 is entered; otherwise, it does not match, and step 5 is entered; Step 5: The Chinese herbal medicine pieces images whose similarity with the standard image features of the Chinese herbal medicine pieces standard images in the cloud data server is within the similarity threshold and the image similarity range are candidate Chinese herbal medicine pieces images, and a candidate Chinese herbal medicine pieces image set is obtained. According to the candidate Chinese herbal medicine pieces image set, a matching image of the Chinese herbal medicine pieces image in the recognition instruction is obtained, and the corresponding Chinese herbal medicine pieces information is obtained according to the matching image, and the identified Chinese herbal medicine pieces information is returned to the client, and step 6 is entered; Step six, complete the identification of Chinese herbal medicine information.

[0006] Furthermore, the recognition task management module obtains the recognition instruction type according to the recognition instruction received from the client, including: Obtain the information in the recognition instruction sent by the client. If the recognition instruction only includes the image information of the Chinese herbal medicine slices or the text information of the Chinese herbal medicine slices, it is a single recognition instruction; otherwise, it is a mixed recognition instruction.

[0007] Furthermore, the identification container corresponding to the identification information matches the identification result information corresponding to the identification information on the cloud data server according to the identification information, including: According to the image information of the Chinese herbal medicine or the text information of the Chinese herbal medicine in the recognition instruction, the text information of the Chinese herbal medicine corresponding to the image information of the Chinese herbal medicine, or the image information of the Chinese herbal medicine corresponding to the text information of the Chinese herbal medicine is matched in the cloud data server.

[0008] Furthermore, the image processing container obtains image features of the Chinese herbal medicine pieces according to the image of the Chinese herbal medicine pieces in the recognition instruction sent by the client, including: The image features are preset image recognition features, and the image recognition features include one or more of the color features, shape features, and texture features of the Chinese herbal medicine pieces.

[0009] Furthermore, the natural language processing container parses the text information of the Chinese herbal medicine pieces in the recognition instruction sent by the client to obtain text features of the Chinese herbal medicine pieces, including: If the text information is the proper name of a Chinese herbal medicine, the text feature of the Chinese herbal medicine is the proper name of the Chinese herbal medicine; if the text information is an alias of a Chinese herbal medicine, the corresponding proper name of the Chinese herbal medicine is obtained according to the alias of the Chinese herbal medicine, and the corresponding proper name of the Chinese herbal medicine is the text feature of the Chinese herbal medicine.

[0010] Furthermore, the method of obtaining a matching image of the Chinese herbal medicine piece image in the recognition instruction based on the candidate Chinese herbal medicine piece image set includes: According to the candidate Chinese herbal medicine decoction piece image set, the similarities between the image features of the Chinese herbal medicine decoction piece image in the recognition instruction and each candidate Chinese herbal medicine decoction piece image in the candidate Chinese herbal medicine decoction piece image set are obtained respectively, among which the Chinese herbal medicine decoction piece image corresponding to the maximum similarity is the matching image of the Chinese herbal medicine decoction piece image in the recognition instruction.

[0011] Furthermore, the method of obtaining corresponding Chinese herbal medicine information according to the matching image and returning the identified Chinese herbal medicine information to the client includes: The text information of the Chinese herbal medicine pieces in the instruction information is corrected to the identified Chinese herbal medicine pieces information, and returned to the client together with the Chinese herbal medicine pieces image information.

[0012] The interactive system for identifying Chinese herbal medicine pieces based on image recognition and natural language applies the Chinese herbal medicine piece identification method based on image recognition and natural language, and includes a cloud data server, a client, and a communication module; the client is connected to the cloud data server through the communication module.

[0013] The present invention has the following beneficial effects: by integrating image recognition and natural language processing technologies, the present invention achieves multi-dimensional recognition of Chinese herbal medicine pieces. Compared with a single recognition method, the present invention can more accurately identify Chinese herbal medicine pieces and reduce recognition errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 The figure is a flowchart of the method for identifying Chinese herbal medicine pieces based on image recognition and natural language; Figure 2 This is a schematic diagram of the principle of the interactive system for identifying Chinese herbal medicine pieces based on image recognition and natural language; Figure 3 This is a schematic diagram of the client principle; Figure 4 Schematic diagram of the user interaction process flow. DETAILED DESCRIPTION

[0015] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0016] The features and performance of the present invention are further described in detail below with reference to the embodiments.

[0017] like Figure 1 As shown, the method for identifying Chinese herbal medicine pieces based on image recognition and natural language includes the following steps: Step 1: The recognition task management module obtains the recognition instruction type based on the recognition instruction received from the client. If it is a single recognition instruction, it proceeds to step 2; if it is a mixed recognition instruction, it proceeds to step 3; In step 2, based on the identification information in the identification instruction, the identification module generates an identification container corresponding to the identification information. The identification container corresponding to the identification information matches the identification result information corresponding to the identification information on the cloud data server based on the identification information, and returns the identification result information corresponding to the identification information to the client, and then proceeds to step 6. Step 3: The recognition module generates an image processing container and a natural language processing container, which are connected to the client respectively. The image processing container obtains the image features of the Chinese herbal medicine slices according to the image of the Chinese herbal medicine slices in the recognition instruction sent by the client, and then proceeds to step 4. At the same time, the natural language processing container parses the text information of the Chinese herbal medicine slices in the recognition instruction sent by the client to obtain the text features of the Chinese herbal medicine slices, and then proceeds to step 4. Step 4: According to the obtained text features of the Chinese herbal medicine pieces, a standard image of the Chinese herbal medicine pieces corresponding to the text features of the Chinese herbal medicine pieces is matched on the cloud data server, the standard image features of the standard image of the Chinese herbal medicine pieces are obtained, and the image similarity with the image features of the image of the Chinese herbal medicine pieces in the recognition instruction is obtained. If the image similarity is greater than the set similarity threshold, the text information of the Chinese herbal medicine pieces in the recognition instruction matches the image of the Chinese herbal medicine pieces in the recognition instruction, and the corresponding recognition result information is returned to the client, and step 6 is entered; otherwise, it does not match, and step 5 is entered; Step 5: The Chinese herbal medicine pieces images whose similarity with the standard image features of the Chinese herbal medicine pieces standard images in the cloud data server is within the similarity threshold and the image similarity range are candidate Chinese herbal medicine pieces images, and a candidate Chinese herbal medicine pieces image set is obtained. According to the candidate Chinese herbal medicine pieces image set, a matching image of the Chinese herbal medicine pieces image in the recognition instruction is obtained, and the corresponding Chinese herbal medicine pieces information is obtained according to the matching image, and the identified Chinese herbal medicine pieces information is returned to the client, and step 6 is entered; Step six, complete the identification of Chinese herbal medicine information.

[0018] The recognition task management module obtains the recognition instruction type according to the recognition instruction received from the client, including: Obtain the information in the recognition instruction sent by the client. If the recognition instruction only includes the image information of the Chinese herbal medicine slices or the text information of the Chinese herbal medicine slices, it is a single recognition instruction; otherwise, it is a mixed recognition instruction.

[0019] The identification container corresponding to the identification information matches the identification result information corresponding to the identification information on the cloud data server according to the identification information, including: According to the image information of the Chinese herbal medicine or the text information of the Chinese herbal medicine in the recognition instruction, the text information of the Chinese herbal medicine corresponding to the image information of the Chinese herbal medicine, or the image information of the Chinese herbal medicine corresponding to the text information of the Chinese herbal medicine is matched in the cloud data server.

[0020] The image processing container obtains image features of the Chinese herbal medicine pieces according to the image of the Chinese herbal medicine pieces in the recognition instruction sent by the client, including: The image features are preset image recognition features, and the image recognition features include one or more of the color features, shape features, and texture features of the Chinese herbal medicine pieces.

[0021] The natural language processing container parses the text information of the Chinese herbal medicine pieces in the recognition instruction sent by the client to obtain text features of the Chinese herbal medicine pieces, including: If the text information is the proper name of a Chinese herbal medicine, the text feature of the Chinese herbal medicine is the proper name of the Chinese herbal medicine; if the text information is an alias of a Chinese herbal medicine, the corresponding proper name of the Chinese herbal medicine is obtained according to the alias of the Chinese herbal medicine, and the corresponding proper name of the Chinese herbal medicine is the text feature of the Chinese herbal medicine.

[0022] The method of obtaining a matching image of the Chinese herbal medicine piece image in the recognition instruction based on the candidate Chinese herbal medicine piece image set includes: According to the candidate Chinese herbal medicine decoction piece image set, the similarities between the image features of the Chinese herbal medicine decoction piece image in the recognition instruction and each candidate Chinese herbal medicine decoction piece image in the candidate Chinese herbal medicine decoction piece image set are obtained respectively, among which the Chinese herbal medicine decoction piece image corresponding to the maximum similarity is the matching image of the Chinese herbal medicine decoction piece image in the recognition instruction.

[0023] The method of obtaining the corresponding Chinese herbal medicine pieces information according to the matching image and returning the identified Chinese herbal medicine pieces information to the client includes: The text information of the Chinese herbal medicine pieces in the instruction information is corrected to the identified Chinese herbal medicine pieces information, and returned to the client together with the Chinese herbal medicine pieces image information.

[0024] like Figure 2 As shown, the interactive system for identifying Chinese herbal medicine pieces based on image recognition and natural language applies the Chinese herbal medicine piece identification method based on image recognition and natural language, including a cloud data server, a client, and a communication module; the client is connected to the cloud data server through the communication module.

[0025] like Figure 3 As shown, the client includes an image acquisition module, a text information input module, a data processing module, a mobile communication device and a display module; The image acquisition module, text information input module, mobile communication device and display module are respectively connected to the data processing module; the mobile communication device is communicatively connected to the communication module.

[0026] Specifically, the present invention proposes a method for identifying Chinese herbal medicine slices based on image recognition and natural language processing. This method achieves efficient and accurate identification of Chinese herbal medicine slices by combining image recognition technology and natural language processing technology. Specifically, the method comprises the following steps: Step 1: Identify instruction management The recognition task management module is responsible for receiving recognition commands sent by the client and determining their type based on the command content. If a recognition command contains only image information or text information about a TCM decoction piece, it is considered a single recognition command; if it contains both, it is considered a mixed recognition command. This step is the foundation for subsequent processing, ensuring that the system selects the appropriate recognition process based on the command type.

[0027] Step 2: Single recognition processing For a single recognition command, the recognition module generates an identification container corresponding to the identification information. This container matches the corresponding recognition result information in the cloud data server based on the identification information (either a TCM decoction piece image or text information). For example, if the identification information is a TCM decoction piece image, the server will return the corresponding TCM decoction piece name, efficacy, and other text information; and vice versa. Once the match is complete, the recognition result information is returned to the client.

[0028] Step 3: Hybrid recognition preprocessing For hybrid recognition instructions, the recognition module simultaneously generates an image processing container and a natural language processing container, each of which is connected to the client. The image processing container extracts image features of TCM pieces, such as color, shape, and texture; the natural language processing container parses the textual information of TCM pieces and extracts textual features, such as the correct name of the TCM piece or the correct name converted from an alias.

[0029] Step 4: Matching text and images After obtaining the image and text features of a TCM piece, the system first searches the cloud data server for a standard image of the TCM piece that matches the text features and retrieves its standard image features. Next, it calculates the image similarity between the TCM piece image in the recognition instruction and the standard image. If the similarity exceeds a set similarity threshold, the text and image are considered a match, and the corresponding recognition result information is returned to the client.

[0030] Step 5: Candidate matching and correction If the image similarity doesn't reach the threshold, the system searches the cloud data server for images of Chinese herbal medicine pieces whose feature similarity to the standard image falls within the threshold and the current similarity, forming a candidate set of Chinese herbal medicine pieces. The system then performs a further match between the candidate set and the image in the recognition instruction to find the most similar matching image, and based on this, retrieves the correct Chinese herbal medicine piece information. Finally, the text in the instruction is modified to reflect the recognized Chinese herbal medicine piece information and returned to the client along with the image information.

[0031] Step 6: Identification completed After the above steps, the system completes the recognition of Chinese herbal medicine information and feeds back the results to the client for user reference or use.

[0032] The recognition task management module parses the recognition command sent by the client to determine whether it contains both TCM decoction piece image information and text information. If it contains only one of these, it is considered a single recognition command; if it contains both, it is considered a mixed recognition command. This judgment process is based on analysis of the command format and content, ensuring accurate identification of the command type.

[0033] For a single recognition command, the recognition container matches the identification information in the cloud data server. The server stores a large number of images and text information of Chinese herbal medicine pieces, as well as the corresponding relationships between them. The recognition container compares the identification information with the data in the server, finds the most matching recognition result information, and returns it to the client.

[0034] The image processing container uses advanced image recognition technologies, such as convolutional neural networks (CNNs), to extract features from images of TCM pieces. These features include color (such as hue, saturation, and brightness), shape (such as outline, edge, and area), and texture (such as roughness and smoothness). These features comprehensively reflect the appearance characteristics of TCM pieces, providing strong support for subsequent matching.

[0035] The natural language processing container is responsible for parsing the textual information of TCM decoction pieces and extracting their textual features. For TCM decoction pieces, the official name is directly used as a textual feature; for aliases, a preset alias-official name mapping table is used to convert them into the corresponding official name, which is then used as a textual feature. This step ensures the accuracy and consistency of the textual information, laying the foundation for subsequent matching and recognition.

[0036] If the image similarity score falls below the threshold, the system searches the cloud data server for images of Chinese herbal medicine pieces with similar features to the standard image, forming a candidate image set. It then uses image matching algorithms (such as SIFT and SURF) to calculate the similarity between the image in the recognition instruction and the candidate image set, finding the most similar matching image. Finally, based on the Chinese herbal medicine piece information corresponding to the matching image, the text in the instruction is corrected to ensure the accuracy of the recognition result.

[0037] The client system includes an image acquisition module, a text information input module, a data processing module, a mobile communication device, and a display module. The image acquisition module is responsible for collecting image information of Chinese herbal medicine pieces; the text information input module allows users to input text information of Chinese herbal medicine pieces; the data processing module is responsible for processing image and text information and communicating with the cloud server; the mobile communication device realizes data transmission between the client and the server; the display module is used to display the recognition results and related information Example 1 The user, a traditional Chinese medicine practitioner, encountered a herbal medicine with an unfamiliar appearance and an unidentified name. The user took a photo of the herbal medicine using the client's image acquisition module and entered the unidentified name (alias) of the herbal medicine using the text input module.

[0038] After receiving the instruction containing image and text information, the recognition task management module determines that it is a mixed recognition instruction and enters the mixed recognition process.

[0039] Hybrid recognition preprocessing: The image processing container uses a convolutional neural network (CNN) to extract features from the captured images of Chinese herbal medicine pieces, and obtains the color, shape, and texture features of the Chinese herbal medicine pieces.

[0040] At the same time, the natural language processing container parses the text information entered by the user and converts the alias into the corresponding correct name of the Chinese herbal medicine as a text feature by querying the preset alias-correct name mapping table.

[0041] Text matching with images: The system searches for standard images of Chinese herbal medicines that match the text features (i.e., the converted corrected names) in the cloud data server and obtains their standard image features.

[0042] Next, the system calculated the image similarity between the user's image of the Chinese herbal medicine piece and the standard image. Due to the uniqueness of this Chinese herbal medicine piece, the similarity during the initial matching did not reach the set threshold.

[0043] Candidate matching and correction: The system searches for images of Chinese herbal medicine pieces whose similarity with the standard image features is within the range of the threshold and the current similarity in the cloud data server, and forms a set of candidate Chinese herbal medicine pieces images.

[0044] Using image matching algorithms (such as SIFT), the system further matches the candidate image set and finally finds the matching image that is most similar to the image taken by the user.

[0045] Based on the Chinese herbal medicine information corresponding to the matching image, the system corrects the text information entered by the user and confirms the correct name of the Chinese herbal medicine and other relevant information.

[0046] Identification completed: The system returns the recognition results (including the corrected text information and original image information) to the client and displays them to the user through the display module.

[0047] Example 2 The user is a salesperson in a Chinese medicine store. When serving a customer, the customer asked for the specific information of a Chinese herbal medicine, but only provided a photo of the Chinese herbal medicine without any text information.

[0048] The salesperson used the client's image acquisition module to take photos of the Chinese herbal medicine pieces provided by the customer and sent a recognition instruction.

[0049] After receiving the instruction containing only image information, the recognition task management module determines that it is a single recognition instruction and enters the single recognition process.

[0050] Single recognition processing: The recognition module generates an identification container corresponding to the identification information and matches it in the cloud data server based on the captured image of the Chinese herbal medicine.

[0051] The server compares the image features to find the record that best matches the image of the Chinese herbal medicine piece and returns the corresponding Chinese herbal medicine piece name, efficacy, and other text information. The server includes a Chinese herbal medicine piece recognition model, which is used to identify the Chinese herbal medicine piece.

[0052] Recognition completed: The system returns the recognition results (including the name of the Chinese herbal medicine, efficacy and original image information) to the client and displays them to the salesperson through the display module.

[0053] The specific Chinese herbal medicine slice recognition model is as follows: The multimodal artificial intelligence technology - multi-scale void attention mechanism (MSDA), large language model (LLM) and LangChain retrieval technology are applied to the classification and recognition of Chinese herbal medicine pieces and the intelligent question-answering system of traditional Chinese medicine, realizing the integrated application of small model (MSDA-YOLOv8) and large model (LLM). The user interaction process is as follows: Figure 4 As shown: Images of Chinese herbal medicines are diverse and complex. In view of the differences in morphological characteristics of different Chinese herbal medicines and the interference caused by complex backgrounds, the project will introduce the MSDA mechanism to improve the recognition accuracy of the YOLOv8 model.

[0054] Multi-scale Dilated Transformer, or DilateFormer for short, is used for visual recognition tasks. Based on this, this scheme proposes a new attention mechanism - Multi-Scale Dilated Attention (MSDA). MSDA can effectively simulate local and sparse image patch interactions within small scales, thereby significantly reducing the redundancy problem commonly found in traditional global attention modules. At the shallow stage of image processing, the attention matrix exhibits two core properties: locality and sparsity. These properties reveal a key fact: in the early stages of semantic modeling, areas far away from the query image patch are generally not directly related to the query patch, so the global attention mechanism often generates a large amount of unnecessary computational redundancy when processing. By introducing the MSDA mechanism, limited computing resources can be used more efficiently, focusing on image areas that are truly relevant to the query patch, thereby achieving more accurate and efficient image processing.

[0055] The MSDA model skillfully strikes a balance between computational complexity and wide receptive field. Through its unique multi-scale dilated attention mechanism, it accurately captures the intricate semantic information in images. A key innovation of the model lies in allowing different attention heads to set different dilation rates as needed, thereby achieving multi-scale feature representation and adapting to details and structures at different scales in the image. MSDA introduces the Sliding Window Dilated Attention (SWDA) mechanism, an efficient and selective method for filtering sparse key points (keys) and values ​​(values) for self-attention. This strategy not only significantly reduces computational redundancy but also enables the model to capture long-range dependencies in the image, further enhancing its understanding of the overall image structure. Furthermore, the MSDA model employs a pyramidal architecture, cleverly combining the locality and sparsity advantages of convolutional neural networks (CNNs) with the global modeling capabilities of the self-attention mechanism. In the shallow architecture, the MSDA block primarily captures low-level image information, while in the deep architecture, a global Multi-Head Self-Attention (MHSA) block captures more high-level and abstract information. This design not only improves model performance but also makes the model more flexible and efficient in handling complex image tasks.

[0056] The self-attention mechanism is the basis of the MSDA model, which is used to calculate the attention weight of each element in the input sequence to all other elements, thereby capturing global information. The calculation of the self-attention mechanism is shown in Formula 1: (1) in, Represents the query matrix, which represents the characteristics of the current element. Represents the key matrix, which represents the characteristics of all elements. Represents the value matrix, which represents the eigenvalues ​​of all elements. It is the dimension of the key vector, which is used to scale the dot product result to prevent the value from being too large. The similarity between the query and the key is calculated and normalized into a weight using the softmax function. This formula calculates the similarity between the query and the key to obtain the attention weight of each element on other elements, thereby performing a weighted summation of the values ​​to achieve global dependency modeling. The multi-head self-attention mechanism introduces multiple attention heads to capture information from different subspaces, thereby enhancing the expressive power of the model. Its calculation is shown in Formula 2: (2) The calculation of each attention head is as follows: in, are projection matrices for query, key, and value respectively. This formula concatenates the results of multiple attention heads and outputs the projection matrix By performing linear transformation, the multi-head self-attention mechanism can capture more dimensional information in the input sequence, improving the expressiveness and robustness of the model. In MSDA, the expansion attention is introduced by the expansion rate To select sparse keys and values ​​to expand the receptive field, the specific calculation is shown in Formula 4: (4) in, and Indicates the expansion rate In this way, MSDA can capture a wider range of contextual information while maintaining computational efficiency.

[0057] This solution introduces the MSDA attention mechanism into the YOLOv8 head network. By enhancing the perception of local and global features at multiple scales, the MSDA mechanism enables the model to capture key features of Chinese herbal medicine pieces in diverse backgrounds and complex environments. The application of multi-scale atrous convolutions improves the capture of edges and details in images, resolving the issue of traditional YOLOv8's unstable performance when handling objects of varying sizes. Furthermore, the Hybrid Feature Extraction module is combined with MSDA to further enhance the processing of image details. Large-scale training of datasets using deep learning algorithms improves the model's adaptability to varying lighting, resolutions, and backgrounds.

[0058] By constructing a dataset of traditional Chinese medicine pieces containing a variety of complex backgrounds and lighting conditions, we trained and optimized an improved YOLOv8 model. We conducted multiple training and validation tests on the model's performance, tailored to different environmental conditions and medicinal material types, to ensure its robustness and efficiency. We also conducted hyperparameter optimization and experimental validation, employing various data augmentation strategies (such as flipping, rotation, and cropping) to improve the model's generalization capabilities. We also conducted a comprehensive evaluation of the model's performance metrics (such as precision, recall, and false positive rate) to ensure its robustness when processing complex images.

[0059] This project uses 9709 images of Chinese herbal medicines crawled by Python crawler, divides 9204 images into training set and 505 images into test set, sets the maximum number of iterations to 100, and uses YOLOv8 model and MSDA-YOLOv8 model for training respectively to realize the detection, recognition and classification of Chinese herbal medicines.

[0060] LangChain technology was used to establish a knowledge base covering multi-dimensional information on Chinese herbal medicines, including classification, efficacy, indications, and chemical composition. Vectorized representation was used to precisely link image recognition results with knowledge base information, ensuring that recognition results could be quickly linked to relevant Chinese medicine information. Fuzzy matching algorithms were used to enhance query flexibility and accuracy, while data aggregation technology was integrated to integrate dispersed information sources, ensuring that the knowledge base could provide users with comprehensive information services. The knowledge base was constructed by aggregating multi-source data from existing literature and Chinese medicine materials, using data cleaning and standardization techniques to ensure the authority and breadth of the knowledge base content.

[0061] Semantic-based fuzzy matching technology can handle imprecise user queries and efficiently match them using a vector database. We studied the relationship between system response time and different query methods to ensure excellent performance in both keyword and semantic query scenarios. Fuzzy query technology combined with the LangChain search generation mechanism enables efficient response to complex query requirements. A vector database is used to enable efficient retrieval of knowledge items and recognition results. Experimental optimization of the search logic enhances the system's query experience.

[0062] By combining semantic analysis with a knowledge base, the accuracy of generative answers is optimized; context-aware technology is used to enhance the fluency of multi-round user interactions. This research trains a large language model on domain-specific corpora to optimize its understanding of questions in the field of Traditional Chinese Medicine (TCM). A context-aware dynamic generative model is introduced to ensure targeted and coherent answers. By combining LLMs (such as Ernie Bot) with a TCM decoction piece knowledge base, an efficient intelligent question-answering system for TCM decoction pieces is constructed.

[0063] Combining fuzzy query technology with dynamic answer generation optimizes the system's response to different types of queries. Based on domain-specific natural language generation technology, the system's ability to handle complex queries is enhanced. By optimizing the collaborative working mechanism between LLM and the knowledge base, it handles ambiguous or incomplete user queries and generates accurate answers.

[0064] Combining adaptive interface design with visualization technology optimizes user convenience. Visualization technology is used to graphically display image recognition results and present the returned Chinese herbal medicine information in a hierarchical manner, enhancing user understanding of complex information. Gradio is used to build a user interface that supports user interaction with the system through image uploads, text queries, and other methods. Research is also conducted on how to use adaptive interface layout technology to ensure consistent display across different devices.

[0065] Adaptive optimization based on user feedback ensures system usability and responsiveness. We utilize user testing and feedback mechanisms to regularly iterate and optimize the interface design, ensuring it meets the needs of users at all levels. Through human-computer interaction model design and user feedback mechanisms, we continuously optimize the interface operation process and user experience. Based on user feedback, the system optimizes the interface layout and query interaction experience to ensure users can intuitively access the information they need.

Claims

1. A method for identifying Chinese herbal medicine pieces based on image recognition and natural language, characterized in that: The steps include: Step 1: The recognition task management module obtains the recognition instruction type according to the recognition instruction received from the client. If it is a single recognition instruction, it proceeds to step 2; if it is a mixed recognition instruction, it proceeds to step 3; Step 2: According to the identification information in the identification instruction, the identification module generates an identification container corresponding to the identification information. The identification container corresponding to the identification information matches the identification result information of the corresponding identification information in the cloud data server according to the identification information, and returns the identification result information of the corresponding identification information to the client, and enters step 6; Step 3: The recognition module generates an image processing container and a natural language processing container, which are connected to the client respectively. The image processing container obtains the image features of the Chinese herbal medicine pieces according to the image of the Chinese herbal medicine pieces in the recognition instruction sent by the client, and then proceeds to step 4; at the same time, the natural language processing container parses the text information of the Chinese herbal medicine pieces in the recognition instruction sent by the client to obtain the text features of the Chinese herbal medicine pieces, and then proceeds to step 4; Step 4: According to the obtained text features of the Chinese herbal medicine pieces, the standard image of the Chinese herbal medicine pieces corresponding to the text features of the Chinese herbal medicine pieces is matched in the cloud data server, the standard image features of the standard image of the Chinese herbal medicine pieces are obtained, and the image similarity with the image features of the image of the Chinese herbal medicine pieces in the recognition instruction is obtained. If the image similarity is greater than the set similarity threshold, the text information of the Chinese herbal medicine pieces in the recognition instruction matches the image of the Chinese herbal medicine pieces in the recognition instruction, and the corresponding recognition result information is returned to the client, and step 6 is entered; otherwise, it does not match, and step 5 is entered; Step 5: The Chinese herbal medicine pieces images whose similarity with the standard image features of the Chinese herbal medicine pieces standard images in the cloud data server is within the similarity threshold and the image similarity range are candidate Chinese herbal medicine pieces images, and a candidate Chinese herbal medicine pieces image set is obtained. According to the candidate Chinese herbal medicine pieces image set, a matching image of the Chinese herbal medicine pieces image in the recognition instruction is obtained, and the corresponding Chinese herbal medicine pieces information is obtained according to the matching image, and the identified Chinese herbal medicine pieces information is returned to the client, and step 6 is entered; Step six, complete the identification of Chinese herbal medicine information.

2. The method for identifying Chinese herbal medicine pieces based on image recognition and natural language according to claim 1, characterized in that: The recognition task management module obtains the recognition instruction type according to the recognition instruction received from the client, including: The information in the recognition instruction sent by the client is obtained. If the recognition instruction only includes the image information of the Chinese herbal medicine pieces or the text information of the Chinese herbal medicine pieces, it is a single recognition instruction; otherwise, it is a mixed recognition instruction.

3. The method for identifying Chinese herbal medicine pieces based on image recognition and natural language according to claim 2, characterized in that: The identification container corresponding to the identification information matches the identification result information corresponding to the identification information in the cloud data server according to the identification information, including: According to the image information of the Chinese herbal medicines or the text information of the Chinese herbal medicines in the recognition instruction, the text information of the Chinese herbal medicines corresponding to the image information of the Chinese herbal medicines or the image information of the Chinese herbal medicines corresponding to the text information of the Chinese herbal medicines is matched in the cloud data server.

4. The method for identifying Chinese herbal medicine pieces based on image recognition and natural language according to claim 2, characterized in that: The image processing container obtains the image features of the Chinese herbal medicine pieces according to the image of the Chinese herbal medicine pieces in the recognition instruction sent by the client, including: The image features are preset image recognition features, and the image recognition features include one or more of the color features, shape features, and texture features of the Chinese herbal medicine pieces.

5. The method for identifying Chinese herbal medicine pieces based on image recognition and natural language according to claim 4, characterized in that: The natural language processing container parses the text information of the Chinese herbal medicine pieces in the recognition instruction sent by the client to obtain the text features of the Chinese herbal medicine pieces, including: If the text information is the proper name of a Chinese herbal medicine, then the text feature of the Chinese herbal medicine is the proper name of the Chinese herbal medicine; if the text information is an alias of a Chinese herbal medicine, then the corresponding proper name of the Chinese herbal medicine is obtained according to the alias of the Chinese herbal medicine, and the corresponding proper name of the Chinese herbal medicine is the text feature of the Chinese herbal medicine.

6. The method for identifying Chinese herbal medicine pieces based on image recognition and natural language according to claim 5, characterized in that: The method of obtaining the matching image of the Chinese herbal medicine piece image in the recognition instruction according to the candidate Chinese herbal medicine piece image set includes: According to the candidate Chinese herbal medicine image set, the similarities between the image features of the Chinese herbal medicine image in the recognition instruction and each candidate Chinese herbal medicine image in the candidate Chinese herbal medicine image set are obtained respectively, among which the Chinese herbal medicine image corresponding to the maximum similarity is the matching image of the Chinese herbal medicine image in the recognition instruction.

7. The method for identifying Chinese herbal medicine pieces based on image recognition and natural language according to claim 6, characterized in that: The method of obtaining corresponding Chinese herbal medicine pieces information according to the matching image and returning the identified Chinese herbal medicine pieces information to the client includes: The text information of the Chinese herbal medicine pieces in the instruction information is corrected to the identified Chinese herbal medicine pieces information, and returned to the client together with the Chinese herbal medicine pieces image information.

8. A Chinese herbal medicine identification interactive system based on image recognition and natural language, characterized in that: The method for identifying Chinese herbal medicine pieces based on image recognition and natural language according to any one of claims 1 to 7 is applied, comprising a cloud data server, a client, and a communication module; the client is connected to the cloud data server through the communication module.

Images