Herbal medicine multi-mode knowledge enhanced disease reasoning system and method

Abstract

The invention provides a disease reasoning system and method for herbal medicine multi-mode knowledge enhancement, and belongs to the field of computer, artificial intelligence and traditional Chinese medicine disease prediction. The system comprises a graphical interface module, a back-end interface module, a traditional Chinese medicine data set module, a prescription data analysis module and an AI disease prediction module. According to the method, by integrating five modes of a chemical structure, molecular fingerprints, physical and chemical attributes, a molecular graph structure and text description, comprehensive modeling of traditional Chinese medicine disease prediction is realized, and the accuracy and robustness of prediction are remarkably improved; a graphical user interface based on a webpage end is designed, searching, browsing, free combination and real-time disease prediction functions of medicinal materials are achieved, structural storage of the medicinal materials and prescription data is adopted, rapid retrieval and dynamic display are supported, the data access efficiency of the whole system is improved, the use threshold of non-professionals is lowered, and the system is suitable for popularization and application. And an efficient and practical tool is provided for disease reasoning in the field of traditional Chinese medicine.

Description

Technical Field

[0001] This invention relates to the fields of computer, artificial intelligence and traditional Chinese medicine disease prediction technology, and in particular to a disease reasoning system and method enhanced with multimodal knowledge of herbal medicine. Background Technology

[0002] Traditional Chinese medicine (TCM) prescription-based disease reasoning systems and platforms are of great significance for disease diagnosis and treatment planning. However, existing TCM prescription-based disease reasoning systems and platforms have many shortcomings and deficiencies. Regarding data utilization, most existing models rely solely on the relationship between prescription and disease or medicinal material and disease, lacking comprehensive utilization of multimodal information such as chemical structure, molecular properties, and textual descriptions. This results in a single data modality, failing to fully explore the complex relationship between TCM and disease. In terms of user interaction, existing tools are mostly presented in command-line, research script, or plain text format, lacking a visual graphical interface for ordinary users. This makes them difficult for non-professionals to operate, creating a high barrier to entry and limiting the system's popularization and promotion. Regarding data access convenience, while some public databases store raw data, they lack convenient retrieval and combined query functions. Users often need to spend a lot of time and effort manually searching and organizing to obtain multimodal information and related prescriptions, resulting in low data access efficiency.

[0003] A thorough analysis reveals that these shortcomings and deficiencies stem from three main aspects. Firstly, at the technical level, early algorithm architectures had limited support for multimodal data fusion, failing to effectively integrate different types of information and resulting in incomplete information utilization. Secondly, at the design level, researchers focused more on showcasing the technical achievements of the model during development, neglecting the usability and visualization needs of end users, leading to a poor user experience in practical applications. Thirdly, at the data management level, existing resources are mostly stored as static files, lacking a dynamic linkage mechanism with the inference model, hindering efficient collaboration between data and the model.

[0004] Multimodal learning technology can fuse different types of features, such as text, structure, image, and numerical data, in a unified vector space, improving the accuracy and robustness of predictions and providing technical support for solving the problem of insufficient utilization of multimodal information. At the same time, the web-based graphical user interface enables data querying, model calling, and visualization through asynchronous interaction, effectively reducing the difficulty of use for non-professional users and improving the user interaction experience. Summary of the Invention

[0005] The purpose of this invention is to provide a disease reasoning system and method that enhances multimodal knowledge of herbs. By integrating multimodal information of herbs, adopting a front-end and back-end separation architecture, and designing a user-friendly graphical interface, it achieves effective fusion of multimodal information, convenient user interaction, and efficient data access, thereby improving the accuracy and practicality of disease reasoning.

[0006] To achieve the above objectives, this invention proposes a disease reasoning system with enhanced multimodal knowledge of herbal medicine, including a graphical interface module, a backend interface module, a traditional Chinese medicine dataset module, a prescription data analysis module, and an AI disease prediction module. The graphical interface module includes a front-end interface and a back-end route. The front-end interface includes a search bar at the top, a selection bar at the bottom left, and a results bar at the bottom right. The backend interface module establishes communication connections with the graphical interface module and the traditional Chinese medicine dataset module; The traditional Chinese medicine dataset module establishes communication connections with the prescription data analysis module and the AI ​​disease prediction module; The prescription data analysis module establishes a communication connection with the backend interface module; The AI ​​disease prediction module is designed based on an attention-based herbal disease prediction model. It includes a multimodal input encoder unit, a multimodal information fusion unit, and a prediction result output unit, and establishes a communication connection with the backend interface module.

[0007] Preferably, the search bar at the top allows users to select search modes, including searching for herbal information, searching for related traditional Chinese medicine prescriptions, and using artificial intelligence models to predict diseases corresponding to herbs. The selection bar at the bottom left allows users to add or delete searched herbal information, and the results bar at the bottom right displays the corresponding search results or prediction results.

[0008] Preferably, the backend interface module adopts an asynchronous separation approach between the frontend and backend. When a retrieval or prediction command is issued, it only waits for the functional module to return the result, requests the query result from the traditional Chinese medicine dataset module, prescription data analysis module, or AI disease prediction module through a routing mechanism, and feeds the result back to the graphical interface module.

[0009] Preferably, in the AI ​​disease prediction module, the multimodal input encoder unit includes a bidirectional recurrent neural network Bi-LSTM, a multilayer neural network perceptron MLP, a graph convolutional neural network GCN, and a scientific domain text encoder SCI-Bert; the multimodal information fusion unit includes a multilayer attention subunit.

[0010] Preferably, the graphical interface module is developed based on JavaScript+HTML technology, the backend interface module is developed based on Python-Flask technology, the traditional Chinese medicine dataset module is developed based on Python-Pandas technology, the prescription data analysis module is developed based on Python technology, and the AI ​​disease prediction module is developed based on Python-PyTorch technology.

[0011] This invention also provides a disease reasoning method enhanced with multimodal knowledge of herbal medicine, comprising the following steps: Step S1: Data input. The user double-clicks to run the executable file, selects the search mode through the search bar of the graphical interface module, enters the Chinese, English or Latin name of the medicinal material in the search bar, selects the column in the lower left corner to add or delete the retrieved herbal information, and adds the selected medicinal material to the current prescription list. Step S2: Collect textual descriptions of herbs, process the SMILES formulas of the corresponding active ingredients of herbs to obtain the corresponding molecular structure diagrams, molecular physicochemical properties, and molecular fingerprint properties, and input the SMILES formulas, molecular structure diagrams, molecular physicochemical properties, molecular fingerprint properties, and textual descriptions of drug properties after atomization and decomposition into a multimodal input encoder; Step S3: Extract multimodal data features and embed the features of the herbal modality respectively; Step S4: Apply a multimodal attention mechanism to fuse the embedding representations of herbal modal features to obtain unique embedding representations of the effective components of herbs. Then, perform linear averaging fusion of the unique embedding representations of the effective components of herbs in the prescription to obtain the embedding representation of the prescription content. Step S5: Based on the embedded representation of the prescription content, predict the probability of each disease. Output the prescription information or AI disease prediction results according to the customer's selection and display them in the results column at the bottom right.

[0012] Preferably, in step S3, a bidirectional recurrent neural network is used to process the atomized SMILES formula to obtain the embedded representation of the atomized SMILES formula; a graph neural network is used to process the molecular graph structure to obtain the embedded representation of the molecular graph structure; a multilayer perceptron is used to process the tensors corresponding to the molecular physicochemical properties and molecular fingerprints to obtain the embedded representations of the tensors corresponding to the molecular physicochemical properties and molecular fingerprints respectively; and a scientific domain text encoder is used to process the herbal text description to obtain the embedded representation of the herbal text description.

[0013] Preferably, in step S4, an attention mechanism is used to perform weighted fusion of the embedding representations of the herbal modal features to obtain a unique embedding representation representing the effective components of the herbal medicine.

[0014] Preferably, in step S5, a multi-layer attention module is used to process the embedded representation information of the prescription content, and the predicted probability corresponding to each disease is calculated through an activation function, and the disease with the higher probability is returned.

[0015] Preferably, in step S5, when the user selects to query a traditional Chinese medicine dataset, the graphical interface module sends a request to the backend interface module, which routes the request to the prescription data analysis module. The prescription data analysis module returns a dataset prescription containing the corresponding traditional Chinese medicine, which is then fed back to the graphical interface module via the backend interface module and displayed in the results column at the bottom right. When the user selects disease prediction, the graphical interface module sends a request to the backend interface module, which routes the request to the AI ​​disease prediction module for processing. The resulting disease prediction is then fed back to the graphical interface module via the backend interface module and displayed in the results column at the bottom right.

[0016] Therefore, this invention proposes a disease reasoning system and method with enhanced multimodal knowledge of herbal medicine, the beneficial effects of which are as follows: (1) The multimodal artificial intelligence model constructed in this invention integrates five modal information: textual description of herbs, chemical molecular SMILES formula, molecular graph structure, molecular physicochemical properties and molecular fingerprint. Through the fusion of multimodal feature embedding and attention mechanism, the value of modal information is fully explored in a unified vector space, effectively capturing the complex relationship between herbs and diseases, significantly improving the accuracy and robustness of disease prediction, and is at the forefront of the field of disease prediction in traditional Chinese medicine.

[0017] (2) The present invention designs a web-based graphical user interface with an intuitive partition design, supports multilingual search for medicinal materials, and allows users to perform medicinal material search, prescription query and disease prediction through simple operations. At the same time, the software is in the form of an executable file, which does not require environment configuration, greatly reducing the difficulty of use for non-professionals, and enabling more users to conveniently use the framework to perform disease reasoning work related to traditional Chinese medicine.

[0018] (3) Through semantic alignment and attention mechanisms, this invention can not only predict diseases, but also analyze the similarity between medicinal materials, trace the source of the prediction results, and enhance the interpretability of the output results.

[0019] (4) The traditional Chinese medicine dataset of the present invention adopts a structured storage method, combined with the routing mechanism of the back-end interface module and the asynchronous interaction technology between the front-end and back-end, to realize the rapid retrieval and dynamic display of traditional Chinese medicine data and prescription data. It can obtain the required multimodal information and prescription data in real time, reduce the time for manual search, and improve the data access efficiency. In addition, asynchronous interaction ensures that the system does not block communication during the process of processing requests, improves the stability and response speed of the system, and improves the user interaction experience. Attached Figure Description

[0020] Figure 1 A structural framework diagram of a disease reasoning system enhanced with multimodal knowledge of herbal medicine; Figure 2A schematic diagram of a graphical interface module for a disease reasoning system enhanced with multimodal knowledge of herbal medicine; Figure 3 This is a schematic diagram of the model structure included in the AI ​​disease prediction module; Figure 4 A flowchart for a disease reasoning method that enhances multimodal knowledge of herbal medicine; Figure 5 A diagram illustrating search results for the herb "Pinellia ternata" based on herbal medicine search terms; Figure 6 This is a diagram showing the search results for the traditional Chinese medicine prescription "donkey-hide gelatin, licorice, and pinellia". Figure 7 This is a schematic diagram of the AI ​​disease prediction results for the traditional Chinese medicine prescription "donkey-hide gelatin, licorice, and pinellia". Detailed Implementation

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0023] Example 1 like Figure 1 As shown, the present invention provides a disease reasoning system with multimodal knowledge enhancement of herbal medicine. The system adopts a front-end and back-end separation architecture, including a graphical interface module, a back-end interface module, a traditional Chinese medicine dataset module, a prescription data analysis module, and an AI disease prediction module. The graphical interface module includes a front-end interface and a back-end route. The front-end interface includes a search bar at the top, a selection bar at the bottom left, and a results bar at the bottom right. The backend interface module establishes communication connections with the graphical interface module and the traditional Chinese medicine dataset module; The traditional Chinese medicine dataset module establishes communication connections with the prescription data analysis module and the AI ​​disease prediction module; The prescription data analysis module establishes a communication connection with the backend interface module; The AI ​​disease prediction module is designed based on an attention-based herbal disease prediction model. It includes a multimodal input encoder unit, a multimodal information fusion unit, and a prediction result output unit, and establishes a communication connection with the backend interface module.

[0024] like Figure 2As shown, the graphical user interface (GUI) module provides a visual operation interface, developed based on JavaScript and HTML technology. The search bar at the top supports three search modes: retrieving herbal information, retrieving related traditional Chinese medicine prescriptions, and using an artificial intelligence model to predict the corresponding diseases associated with herbs. It also supports searching for herbs in Chinese, English, and Latin. The selection bar at the bottom left allows users to add or delete retrieved herbal information; only herbs listed in this bar will be used in subsequent prescription searches or disease predictions. The results bar at the bottom right displays the corresponding search results or AI disease prediction results for easy and intuitive viewing. This module interacts with the backend interface module, sending user operation requests to the backend and receiving and displaying the results from the backend.

[0025] The backend interface module acts as a bridge for communication between the frontend and backend. It receives requests from the graphical interface module, distributes the requests to the corresponding functional modules (such as the traditional Chinese medicine dataset module, prescription data analysis module, or AI disease prediction module) through a routing mechanism, requests query results, and feeds the results back to the graphical interface module. This module is developed based on Python-Flask technology and adopts an asynchronous separation approach between the frontend and backend. When a retrieval or prediction command is issued, it only waits for the functional module to return the result without blocking communication, ensuring that the system has good stability and response speed.

[0026] The Traditional Chinese Medicine Dataset Module uses a structured storage method to store herb-related data, including textual descriptions of herbs and chemical molecular information of active ingredients, such as SMILES formulas, molecular diagram structures, molecular physicochemical properties, and molecular fingerprints. When a query request is received from the backend interface module, this module quickly retrieves relevant data based on the request content and returns the herb query results and detailed information about the herbs, providing users with comprehensive TCM data support. This module is developed based on Python-Pandas technology.

[0027] The prescription data analysis module stores prescription datasets containing different combinations of herbs. When the backend interface module forwards a user's prescription query request, this module retrieves the prescription dataset containing these herbs based on the herb information specified in the request and returns the results to the backend interface module, helping users understand the relevant prescription information. It is developed based on Python technology.

[0028] like Figure 3 As shown, the AI ​​disease prediction module is the core module for implementing disease prediction functions. Based on a multimodal artificial intelligence model, it returns the actual diseases that corresponding herbal remedies may address based on requests from the backend interface module. This module is developed using Python-PyTorch technology and specifically includes: The multimodal input encoder unit collects textual descriptions of relevant herbs, processes the SMILES formulas of the corresponding active ingredients of the herbs to obtain molecular graph structures, molecular physicochemical properties, and molecular fingerprints, and performs atomization decomposition of the SMILES formulas for later use. The multimodal information fusion unit and the multimodal input encoder unit include a bidirectional recurrent neural network Bi-LSTM, a multilayer neural network perceptron MLP, a graph convolutional neural network GCN, and a scientific domain text encoder SCI-Bert. The multimodal information fusion unit comprises multiple attention subunits. It employs a bidirectional recurrent neural network to process atomized SMILES structures to obtain embedded representations, a graph neural network to process molecular graph structures to obtain embedded representations, a multilayer perceptron to process the tensors corresponding to molecular physicochemical properties and molecular fingerprints to obtain self-embedded representations, and a scientific domain text encoder to process herbal text descriptions to obtain embedded representations. Then, an attention mechanism is used to weightedly fuse the herbal modal embedded representations to obtain unique embedded representations representing the effective components of the herbs. The embedded representations of the herbs in the prescription are then linearly averaged and fused to obtain the embedded representation of the prescription content. Finally, a multilayer attention module processes the prescription embedded representation information, and an activation function is used to calculate the predicted probability for each disease, returning the disease with the higher probability. The prediction result output unit outputs the prediction result and transmits the result to the backend interface module.

[0029] Example 2 like Figure 4 As shown, the present invention also provides a disease reasoning method enhanced with multimodal knowledge of herbal medicine, comprising the following steps: Step S1: Data input. The user double-clicks to run the executable file, selects the search mode through the search bar of the graphical interface module, enters the Chinese, English or Latin name of the medicinal material in the search bar, selects the column in the lower left corner to add or delete the retrieved herbal information, and adds the selected medicinal material to the current prescription list. Step S2: Collect textual descriptions of herbs, process the SMILES formulas of the corresponding active ingredients of herbs to obtain the corresponding molecular structure diagrams, molecular physicochemical properties, and molecular fingerprint properties, and input the SMILES formulas, molecular structure diagrams, molecular physicochemical properties, molecular fingerprint properties, and textual descriptions of drug properties after atomization and decomposition into a multimodal input encoder; Step S3: Extract multimodal data features and embed the features of the herbal modality into the following ways: use a bidirectional recurrent neural network to process the atomized SMILES formula to obtain the embedding representation of the atomized SMILES formula; use a graph neural network to process the molecular graph structure to obtain the embedding representation of the molecular graph structure; use a multilayer perceptron to process the tensors corresponding to the molecular physicochemical properties and molecular fingerprints to obtain the embedding representations of the tensors corresponding to the molecular physicochemical properties and molecular fingerprints; and use a scientific domain text encoder to process the herbal text description to obtain the embedding representation of the herbal text description. Step S4: Apply a multimodal attention mechanism to fuse the embedding representations of herbal modal features. Specifically, use the attention mechanism to perform weighted fusion of the embedding representations of herbal modal features to obtain a unique embedding representation representing the effective components of the herbs. Then, perform linear average fusion of the unique embedding representations of the effective components of the herbs contained in the prescription to obtain the embedding representation of the prescription content. Step S5: Based on the embedding table of the prescription content, predict the probability of each disease. Output the prescription information or the AI ​​disease prediction result according to the customer's selection, and display it in the results column at the bottom right. Specifically: Use a multi-layer attention module to process the prescription embedding representation information, calculate the predicted probability of each disease through an activation function, and return the disease with the higher probability. When a user selects to query a Chinese herbal medicine dataset, the graphical interface module sends a request to the backend interface module. The backend interface module routes the request to the prescription data analysis module, which returns a dataset prescription containing the corresponding Chinese herbal medicine. This prescription is then fed back to the graphical interface module via the backend interface module and displayed in the results section at the bottom right. When a user selects disease prediction, the graphical interface module sends a request to the backend interface module. The backend interface module routes the request to the AI ​​disease prediction module for processing, obtains the disease prediction result, and then feeds it back to the graphical interface module via the backend interface module. The AI ​​disease prediction result is displayed in the results column at the bottom right.

[0030] The invention will be further illustrated below through specific implementation examples.

[0031] The specific implementation example of this invention is the herb "Pinellia ternata". The information is entered into the system for querying.

[0032] Step S1: Double-click to run the executable file. Select the search mode "Search by Herb", "Search by Prescription" or "Al Symptom Prediction" through the search bar of the graphical interface module. Enter the Chinese name of the medicinal material, Banxia, ​​in the search bar. The prescription data analysis module will retrieve the prescription dataset containing the combination of "Banxia", "Ejiao", and "Gancao" and return the results to the graphical interface module. The results will be displayed in the results bar. Users can view detailed prescription information and select the column in the lower left corner to add or delete the retrieved herbal information. Add all medicinal materials to the current prescription list. Step S2: Collect textual descriptions of herbs, process the SMILES formulas of the corresponding active ingredients of herbs to obtain the corresponding molecular structure diagrams, molecular physicochemical properties, and molecular fingerprint properties, and input the SMILES formulas, molecular structure diagrams, molecular physicochemical properties, molecular fingerprint properties, and textual descriptions of drug properties after atomization and decomposition into a multimodal input encoder; Step S3: Extract multimodal data features and embed the features of the herbal modality into the following ways: use a bidirectional recurrent neural network to process the atomized SMILES formula to obtain the embedding representation of the atomized SMILES formula; use a graph neural network to process the molecular graph structure to obtain the embedding representation of the molecular graph structure; use a multilayer perceptron to process the tensors corresponding to the molecular physicochemical properties and molecular fingerprints to obtain the embedding representations of the tensors corresponding to the molecular physicochemical properties and molecular fingerprints; and use a scientific domain text encoder to process the herbal text description to obtain the embedding representation of the herbal text description. Step S4: Apply a multimodal attention mechanism to fuse the embedding representations of herbal modal features. Specifically, use the attention mechanism to perform weighted fusion of the embedding representations of herbal modal features to obtain a unique embedding representation representing the effective components of the herbs. Then, perform linear average fusion of the unique embedding representations of the effective components of the herbs contained in the prescription to obtain the embedding representation of the prescription content. Step S5: Based on the embedded representation of the prescription content, predict the probability of each disease. Depending on the customer's selection, output the prescription information or the AI ​​disease prediction result, and display it in the results section at the bottom right. Specifically: Use a multi-layer attention module to process the prescription embedded representation information, calculate the predicted probability of each disease through an activation function, and return the disease with the higher probability. When a user selects to query a traditional Chinese medicine (TCM) dataset, and the search mode is "Search by Herb," ​​the graphical interface module sends a request to the backend interface module. The backend interface module routes the request to the TCM multimodal data module, which returns a dataset containing the corresponding TCM. This dataset is then fed back to the graphical interface module via the backend interface module and displayed in the results section at the bottom right. The results are shown below. Figure 5As shown, the results column in the lower right corner displays the multimodal information of "Pinellia".

[0033] When a user selects to query a traditional Chinese medicine dataset, and the search mode is "Search by Prescription," the graphical interface module sends a request to the backend interface module. The backend interface module routes the request to the prescription data analysis module, which returns a dataset containing the corresponding traditional Chinese medicine. This data is then fed back to the graphical interface module via the backend interface module and displayed in the results section at the bottom right. The results are shown below. Figure 6 As shown, the prescription information containing "Pinellia" is displayed in the results column at the bottom right.

[0034] When a user selects disease prediction and the search mode is "AI Symptom Prediction," the graphical interface module sends a request to the backend interface module. The backend interface module routes the request to the AI ​​disease prediction module for processing, obtains the disease prediction result, and then feeds it back to the graphical interface module. The AI ​​disease prediction result is displayed in the results section at the bottom right, as shown in the image. Figure 7 As shown, the results section displays "AI Symptom Predictions: Predicted: Vomiting blood, Predicted: Vomiting, Predicted: Coughing," and users can view the prediction results.

[0035] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0036] Therefore, this invention provides a disease reasoning system and method enhanced with multimodal knowledge of herbal medicines. By acquiring multimodal data of herbal medicines, it utilizes an attention mechanism to weightedly fuse multimodal embeddings, and calculates the disease prediction probability based on the fused representation using multi-layer attention modules and activation functions. The system adopts a front-end and back-end separation architecture. The front-end provides a graphical interface, while the back-end provides data processing and model reasoning functions. Users can search for herbs, combine prescriptions, query data, or perform disease predictions through the interface. The AI ​​model fuses herbal medicine information through a multimodal attention mechanism and outputs disease prediction results. Compared with traditional prediction models, the prediction accuracy and robustness are improved. The intuitive web-based graphical interface lowers the professional threshold, and structured storage supports fast querying, reducing manual search time and achieving efficient data access, thus enhancing the system's practicality.

[0037] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A disease reasoning system enhanced with multimodal knowledge of herbal medicine, characterized in that: It includes a graphical interface module, a backend interface module, a traditional Chinese medicine dataset module, a prescription data analysis module, and an AI disease prediction module; The graphical interface module includes a front-end interface and a back-end route. The front-end interface includes a search bar at the top, a selection bar at the bottom left, and a results bar at the bottom right. The backend interface module establishes communication connections with the graphical interface module and the traditional Chinese medicine dataset module; The traditional Chinese medicine dataset module establishes communication connections with the prescription data analysis module and the AI ​​disease prediction module; The prescription data analysis module establishes a communication connection with the backend interface module; The AI ​​disease prediction module is designed based on an attention-based herbal disease prediction model. It includes a multimodal input encoder unit, a multimodal information fusion unit, and a prediction result output unit, and establishes a communication connection with the backend interface module.

2. The disease reasoning system with enhanced multimodal knowledge of herbal medicine according to claim 1, characterized in that: The search bar at the top allows you to select search modes, including searching for herbal information, searching for related traditional Chinese medicine prescriptions, and using artificial intelligence models to predict the diseases corresponding to herbs. The selection bar at the bottom left allows you to add or delete searched herbal information, and the results bar at the bottom right displays the corresponding search results or prediction results.

3. The disease reasoning system with enhanced multimodal knowledge of herbal medicine according to claim 1, characterized in that: The backend interface module adopts an asynchronous separation approach between the frontend and backend. When a retrieval or prediction command is issued, it only waits for the functional module to return the result, requests the query result from the traditional Chinese medicine dataset module, prescription data analysis module, or AI disease prediction module through a routing mechanism, and then feeds the result back to the graphical interface module.

4. A disease reasoning system with enhanced multimodal knowledge of herbal medicine as described in claim 1, characterized in that: In the AI ​​disease prediction module, the multimodal input encoder unit includes a bidirectional recurrent neural network Bi-LSTM, a multilayer neural network perceptron MLP, a graph convolutional neural network GCN, and a scientific domain text encoder SCI-Bert; the multimodal information fusion unit includes a multilayer attention subunit.

5. A disease reasoning system with enhanced multimodal knowledge of herbal medicine as described in claim 1, characterized in that: The graphical interface module is developed using JavaScript and HTML technology, the backend interface module is developed using Python-Flask technology, the traditional Chinese medicine dataset module is developed using Python-Pandas technology, the prescription data analysis module is developed using Python technology, and the AI ​​disease prediction module is developed using Python-PyTorch technology.

6. A disease reasoning method enhanced with multimodal knowledge of herbal medicine, characterized in that, Includes the following steps: Step S1: Data input. The user double-clicks to run the executable file, selects the search mode through the search bar of the graphical interface module, enters the Chinese, English or Latin name of the medicinal material in the search bar, selects the column in the lower left corner to add or delete the retrieved herbal information, and adds the selected medicinal material to the current prescription list. Step S2: Collect textual descriptions of herbs, process the SMILES formulas of the corresponding active ingredients of herbs to obtain the corresponding molecular structure diagrams, molecular physicochemical properties, and molecular fingerprint properties, and input the SMILES formulas, molecular structure diagrams, molecular physicochemical properties, molecular fingerprint properties, and textual descriptions of drug properties after atomization and decomposition into a multimodal input encoder; Step S3: Extract multimodal data features and embed the features of the herbal modality respectively; Step S4: Apply a multimodal attention mechanism to fuse the embedding representations of herbal modal features to obtain unique embedding representations of the effective components of herbs. Then, perform linear averaging fusion of the unique embedding representations of the effective components of herbs in the prescription to obtain the embedding representation of the prescription content. Step S5: Based on the embedded representation of the prescription content, predict the probability of each disease. Output the prescription information or AI disease prediction results according to the customer's selection and display them in the results column at the bottom right.

7. The disease reasoning method with enhanced multimodal knowledge of herbal medicine according to claim 6, characterized in that: In step S3, a bidirectional recurrent neural network is used to process the atomized SMILES formula to obtain the embedded representation of the atomized SMILES formula. A graph neural network is used to process the molecular graph structure to obtain the embedded representation of the molecular graph structure. A multilayer perceptron is used to process the tensors corresponding to the molecular physicochemical properties and molecular fingerprints to obtain the embedded representations of the tensors corresponding to the molecular physicochemical properties and molecular fingerprints respectively. A scientific domain text encoder is used to process the herbal text description to obtain the embedded representation of the herbal text description.

8. A disease reasoning method with enhanced multimodal knowledge of herbal medicine as described in claim 6, characterized in that: In step S4, an attention mechanism is used to perform weighted fusion of the embedding representations of herbal modal features to obtain a unique embedding representation representing the effective components of the herbal medicine.

9. A disease reasoning method based on multimodal knowledge enhancement of herbal medicine according to claim 6, characterized in that: In step S5, a multi-layer attention module is used to process the embedded representation information of the prescription content, and the prediction probability corresponding to each disease is calculated through the activation function, and the disease with the higher probability is returned.

10. A disease reasoning method with enhanced multimodal knowledge of herbal medicine as described in claim 6, characterized in that: In step S5, when the user selects to query a traditional Chinese medicine dataset, the graphical interface module sends a request to the backend interface module, which routes the request to the prescription data analysis module. The prescription data analysis module returns a dataset prescription containing the corresponding traditional Chinese medicine, which is then fed back to the graphical interface module via the backend interface module and displayed in the results column at the bottom right. When the user selects disease prediction, the graphical interface module sends a request to the backend interface module, which routes the request to the AI ​​disease prediction module for processing. The backend interface module then feeds back the disease prediction result to the graphical interface module, which displays the AI ​​disease prediction result in the results column at the bottom right.

Images