Methods, systems, equipment, and media for recommending traditional Chinese medicine based on multimodal fusion

Abstract

This application discloses a method, system, device, and medium for TCM recommendation based on multimodal fusion. The method constructs a heterogeneous graph based on core representative objects and preprocessed TCM multimodal data; inputs the heterogeneous graph into a graph neural network for data fusion to obtain TCM multimodal fused data; constructs a knowledge graph based on the TCM multimodal fused data to obtain a TCM knowledge graph; divides the TCM knowledge graph into subgraphs for multi-agent processing according to efficacy and medication safety, obtaining a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent; based on the medicinal material subgraph and the safety subgraph, iteratively optimizes the selection of medicinal materials through a combination strategy and a review strategy for medicinal material safety to obtain the optimal TCM prescription; inputs the optimal prescription into a fine-tuned large language model to determine the target TCM prescription. This application can improve the accuracy of TCM recommendations.

Description

Technical Field

[0001] This application relates to the field of intelligent medical technology, and in particular to a method, system, device and medium for recommending traditional Chinese medicine based on multimodal fusion. Background Technology

[0002] Traditional Chinese medicine (TCM) is centered on holistic concepts and syndrome differentiation and treatment, emphasizing individualized treatment. Its knowledge system encompasses complex relationships across multiple dimensions, including Chinese herbal medicines, prescriptions, symptoms, syndromes, and treatment methods. Traditional TCM recommendations rely heavily on physician experience, leading to low accuracy for inexperienced physicians. To address this issue, existing technologies have proposed several TCM recommendation methods based on machine learning or deep learning.

[0003] Existing TCM recommendation methods have many shortcomings. For example, (1) the TCM recommendation method based on syndrome information using graph neural networks has excessively high computational complexity, making it difficult to apply to scenarios such as real-time recommendation and clinical diagnosis. The model lacks interpretability; the complex structure of graph neural networks and multilayer perceptrons makes the recommendation results difficult to understand and interpret intuitively, which may affect the trust of doctors and patients in the recommendation results. In addition, the model may lack dynamic adaptability, lacking the ability to adapt to dynamically changing TCM diagnosis and treatment data, and cannot be updated and optimized in a timely manner to cope with new symptoms or treatment methods. (2) the TCM recommendation method based on multi-graph convolutional neural networks has excessively high data dependence; this method relies on datasets from classic medical books such as the *Shanghan Lun*, which may not be able to fully cover the complex situations and diverse symptom combinations in modern TCM diagnosis and treatment. Furthermore, the model has excessively high complexity, requiring a large amount of computational resources. At the same time, it lacks the ability to adapt to dynamically changing TCM diagnosis and treatment data, and cannot be updated and optimized in a timely manner to cope with new symptoms or treatment methods.

[0004] In summary, existing methods for recommending TCM have relatively low accuracy and poor adaptability to dynamically changing data. Summary of the Invention

[0005] This application aims to propose a method, system, device, and medium for recommending traditional Chinese medicine based on multimodal fusion, which can improve the accuracy of traditional Chinese medicine recommendations and enhance adaptability to dynamically changing data.

[0006] In a first aspect, embodiments of this application provide a method for recommending traditional Chinese medicine based on multimodal fusion, the method comprising:

[0007] Acquire multimodal data of traditional Chinese medicine that includes text, image, audio, and video data;

[0008] The TCM multimodal data is preprocessed to obtain preprocessed TCM multimodal data;

[0009] Based on the core representative objects and the preprocessed TCM multimodal data, a heterogeneous graph is constructed, wherein the core representative objects include the core medicinal materials in the TCM multimodal data;

[0010] The heterogeneous graph is input into a graph neural network, and the preprocessed TCM multimodal data is fused through the graph neural network to obtain TCM multimodal fused data.

[0011] A knowledge graph of traditional Chinese medicine is constructed based on the aforementioned multimodal fusion data of traditional Chinese medicine.

[0012] The TCM knowledge graph is divided into subgraphs for multi-agent processing according to efficacy and safety, resulting in a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent. The multi-agent includes two types: efficacy agent and safety agent. The efficacy agent includes a combination strategy for medicinal materials, and the safety agent includes a strategy for verifying the safety of medicinal materials.

[0013] Based on the medicinal material subgraph and the safety subgraph, the selection of medicinal materials is optimized through collaborative iterative optimization using the combined medicinal material strategy and the approved medicinal material safety strategy, resulting in the optimal compound prescription recommended by traditional Chinese medicine.

[0014] The optimal compound is input into the finely tuned large language model to determine the target compound recommended by traditional Chinese medicine.

[0015] Compared with the prior art, the first aspect of this application has the following beneficial effects:

[0016] This method acquires multimodal data of Traditional Chinese Medicine (TCM) including text, image, audio, and video data. It preprocesses this data to obtain preprocessed TCM multimodal data. Based on core representative objects and the preprocessed TCM multimodal data, a heterogeneous graph is constructed. The core representative objects include core medicinal materials from the TCM multimodal data. This heterogeneous graph is then input into a graph neural network (Graph Neural Network) to fuse the preprocessed TCM multimodal data, resulting in fused TCM multimodal data. This fused data enriches the information dimensions of TCM data, possesses high flexibility and scalability, and can meet the data application needs of various scenarios. Finally, a knowledge graph is constructed based on the fused TCM multimodal data, resulting in a TCM knowledge graph, which enables the visual representation of TCM data. The TCM knowledge graph is then divided into subgraphs for multi-agent processing based on efficacy and medication safety. This results in a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent. The multi-agent agents include efficacy agents and safety agents. The efficacy agent includes a medicinal material combination strategy, while the safety agent includes a medicinal material safety verification strategy. Based on the medicinal material and safety subgraphs, the selection of medicinal materials is optimized through collaborative iteration using the medicinal material combination strategy and the medicinal material safety verification strategy. This yields the optimal TCM prescription. By decomposing the TCM knowledge graph into subgraphs and corresponding agents, and establishing a dynamic collaboration mechanism between agents, accurate recommendation and safety verification of TCM prescriptions are achieved. Finally, the optimal prescription is input into a fine-tuned large language model to determine the target TCM prescription, generating a complete prescription containing precise dosage, decoction method, and treatment course. This improves the accuracy of TCM recommendations and enhances adaptability to dynamically changing data.

[0017] In some embodiments, the preprocessing of the TCM multimodal data to obtain preprocessed TCM multimodal data includes:

[0018] The text data in the TCM multimodal data is preprocessed using word embedding to obtain the first feature vector;

[0019] A convolutional neural network is used to preprocess the image data in the TCM multimodal data to obtain a second feature vector;

[0020] The audio data in the TCM multimodal data is preprocessed using a deep learning model to obtain a third feature vector;

[0021] A convolutional neural network is used to preprocess the video data in the TCM multimodal data to obtain a fourth feature vector;

[0022] The first feature vector, the second feature vector, the third feature vector, and the fourth feature vector are normalized and dimensionality reduced to obtain preprocessed TCM multimodal data.

[0023] In some implementations, the step of constructing a knowledge graph based on the TCM multimodal fusion data to obtain a TCM knowledge graph includes:

[0024] The core objects in the TCM multimodal data are selected as entity nodes, wherein the core objects include medicinal materials and symptoms in the TCM multimodal data;

[0025] The aforementioned multimodal fusion data of traditional Chinese medicine is used as the attribute information of entity nodes;

[0026] Relationship edges are constructed based on the relationships between the entity nodes, and initial weights are preset for the relationship edges;

[0027] Based on the entity nodes, the attribute information, and the relationship edges, an initial knowledge graph is constructed;

[0028] Markov decision is used to adjust the initial weights of the relation edges in the initial knowledge graph to obtain optimized weights;

[0029] The optimized weights are used to replace the initial weights of the relation edges in the initial knowledge graph to construct a traditional Chinese medicine knowledge graph.

[0030] In some implementations, the step of segmenting the traditional Chinese medicine knowledge graph into subgraphs based on efficacy and medication safety, to obtain a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent, includes:

[0031] The TCM knowledge graph is constructed as a main graph structure containing nodes, binary edges, hyperedges, and node feature vectors.

[0032] The node feature vectors in the main graph structure are learned by multiple agents, and the node feature vectors in the learning process are enhanced by feature projection matrix to obtain the enhanced node feature vectors corresponding to each agent.

[0033] Based on the enhanced node feature vector corresponding to each agent, the similarity between nodes is calculated, and a sparse adjacency matrix corresponding to each agent is constructed based on the similarity.

[0034] Construct a degree matrix based on the sparse adjacency matrix corresponding to each agent;

[0035] Subtracting the degree matrix from the sparse adjacency matrix yields the nonnormalized Laplace matrix.

[0036] Clustering is performed on the eigenvectors in the non-normalized Laplacian matrix to obtain multiple clustering results;

[0037] Each clustering result is processed as a subgraph by an agent to obtain a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent.

[0038] In some implementations, the step of selecting medicinal materials based on the medicinal material subgraph and the safety subgraph through collaborative iterative optimization using the combined medicinal material strategy and the approved medicinal material safety strategy to obtain the optimal compound prescription recommended by traditional Chinese medicine includes:

[0039] Using the aforementioned combined medicinal material strategy, medicinal materials are selected from the medicinal material sub-graph and combined to obtain the first compound prescription;

[0040] Based on the safety subgraph, the medicinal materials in the first compound are subjected to safety audit using the aforementioned medicinal material safety audit strategy to obtain the second compound.

[0041] The combined medicinal material strategy and the approved medicinal material safety strategy are synergistically iteratively optimized until the second compound is the same in two consecutive synergistic iterative optimizations, thus obtaining the optimal compound recommended by traditional Chinese medicine.

[0042] In some embodiments, the herbal sub-graph includes symptom-treatment relationship between herbs, and the combination of herbs from the herbal sub-graph to obtain a first compound formula using the combined herbal strategy includes:

[0043] Based on the patient's symptoms, calculate the comprehensive efficacy score of each herb in the herb sub-graph for the symptoms;

[0044] Iterative adjustments are made based on the combined medicinal materials strategy, including: selecting a preset number of medicinal materials from high to low according to the comprehensive efficacy score to obtain an initial compound prescription;

[0045] The medicinal materials in the initial compound prescription whose comprehensive efficacy score is greater than or equal to the first preset value shall be retained;

[0046] After removing medicinal materials with a comprehensive efficacy score lower than the second preset value from the initial compound prescription, the medicinal materials are replenished to the preset number according to the comprehensive efficacy score from high to low.

[0047] The first compound is obtained when the changes in the medicinal components in the compound are less than a third preset value after two consecutive rounds of iterative adjustments of the combined medicinal material strategy, or when the fluctuation of the total efficacy score of all medicinal materials in the compound is less than a fourth preset value.

[0048] In some implementations, the safety subgraph includes medicinal material entity nodes and incompatibility relationship edges. Based on the safety subgraph, the medicinal material safety audit strategy is used to perform a safety audit on the medicinal materials in the first compound to obtain a second compound, including:

[0049] The safety strategy for reviewing medicinal materials is iteratively adjusted, including: obtaining the single-drug toxicity of each medicinal material and removing medicinal materials in the first compound whose single-drug toxicity is greater than or equal to the first safety threshold.

[0050] Calculate the total toxicity of all medicinal materials in the first compound, and calculate the contribution of each medicinal material to the therapeutic effect. Based on the contribution of the therapeutic effect and the single-drug toxicity, calculate the toxicity efficiency ratio of each medicinal material.

[0051] If the total toxicity is greater than the second safety threshold, then the medicinal material with the low toxicity-efficiency ratio is removed;

[0052] If there is a compatibility incompatibility relationship edge in the first compound, then the medicinal material pair is obtained based on the two medicinal material entity nodes corresponding to the compatibility incompatibility relationship edge;

[0053] Calculate the topological fitness of each herb in the herb pair, and remove the herb with low topological fitness in the herb pair;

[0054] The second compound formula is obtained when the medicinal materials in the compound formula remain unchanged after two consecutive rounds of iterative adjustments to the safety strategy for the medicinal materials being reviewed.

[0055] Secondly, embodiments of this application also provide a traditional Chinese medicine recommendation system based on multimodal fusion, the system comprising:

[0056] The data acquisition unit is used to acquire multimodal data of traditional Chinese medicine, including text data, image data, audio data, and video data.

[0057] The data processing unit is used to preprocess the TCM multimodal data to obtain preprocessed TCM multimodal data.

[0058] The heterogeneous graph construction unit is used to construct a heterogeneous graph based on the core representative objects and the preprocessed TCM multimodal data, wherein the core representative objects include the core medicinal materials in the TCM multimodal data;

[0059] The data fusion unit is used to input the heterogeneous graph into the graph neural network, and to fuse the preprocessed TCM multimodal data through the graph neural network to obtain TCM multimodal fused data.

[0060] The graph construction unit is used to construct a knowledge graph based on the TCM multimodal fusion data to obtain a TCM knowledge graph;

[0061] The graph segmentation unit is used to segment the TCM knowledge graph into subgraphs processed by multiple agents according to efficacy and medication safety, to obtain a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent. The multiple agents include two types: efficacy agents and safety agents. The efficacy agents include a combination strategy for medicinal materials, and the safety agents include a strategy for reviewing the safety of medicinal materials.

[0062] The collaborative optimization unit is used to select medicinal materials through collaborative iterative optimization based on the medicinal material subgraph and the safety subgraph, using the combined medicinal material strategy and the approved medicinal material safety strategy, to obtain the optimal compound prescription recommended by traditional Chinese medicine.

[0063] The compound determination unit is used to input the optimal compound into the fine-tuned large language model to determine the target compound recommended by traditional Chinese medicine.

[0064] Thirdly, embodiments of this application also provide an electronic device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform a traditional Chinese medicine recommendation method based on multimodal fusion as described above.

[0065] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to execute a multimodal fusion-based traditional Chinese medicine recommendation method as described above.

[0066] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0067] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0068] Figure 1 This is a flowchart illustrating an embodiment of the TCM recommendation method based on multimodal fusion provided in this application;

[0069] Figure 2 This is a schematic diagram of the overall method flow in the best embodiment of the TCM recommendation method based on multimodal fusion provided in this application;

[0070] Figure 3This is a schematic diagram of multimodal data preprocessing in the best embodiment of the TCM recommendation method based on multimodal fusion provided in this application;

[0071] Figure 4 This is a flowchart illustrating the TCM multimodal data fusion module in the best embodiment of the TCM recommendation method based on multimodal fusion provided in this application;

[0072] Figure 5 This is a schematic diagram of the knowledge graph construction process based on reinforcement learning in the best embodiment of the TCM recommendation method based on multimodal fusion provided in this application;

[0073] Figure 6 This is a schematic diagram of the two-layer treatment architecture in the best embodiment of the TCM recommendation method based on multimodal fusion provided in this application;

[0074] Figure 7 This is a schematic diagram of an embodiment of the TCM recommendation system based on multimodal fusion provided in this application. Detailed Implementation

[0075] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0076] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0077] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0078] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0079] First, let's analyze some of the terms used in this application:

[0080] Text cleaning technology refers to the preprocessing of raw text data to remove useless information, correct errors, and standardize formatting, thereby improving text quality and making it more suitable for subsequent analysis and processing.

[0081] Principal Component Analysis (PCA): A commonly used dimensionality reduction technique. Its principle is to perform eigenvalue decomposition on the covariance matrix of the data, find the orthogonal directions with the largest variance in the data (i.e., principal components), and then project the original data onto these principal component directions, thereby reducing the dimensionality of the data while retaining as much of the main information of the data as possible.

[0082] Adjacency matrix: An adjacency matrix is ​​a way to represent a graph, using a two-dimensional matrix to represent the connection relationships between nodes in the graph.

[0083] Heterogeneous graph: A heterogeneous graph is a graph in which nodes and edges can be of different types, and there can be specific semantic relationships between different types.

[0084] Degree matrix: The degree matrix is ​​a diagonal matrix associated with a graph, where each diagonal element represents the degree of the corresponding node (i.e., the number of edges connected to the node). In the computation of graph neural networks, the degree matrix is ​​often used to normalize the feature aggregation of nodes, so as to avoid the feature information of some nodes being over-amplified or under-amplified during propagation due to excessive differences in node degree, which helps to improve the stability and performance of the model.

[0085] Traditional Chinese Medicine (TCM) Knowledge Graph: This is a structured knowledge system that integrates data on TCM theories, medicinal materials, and prescriptions. It reveals inherent patterns by establishing relationships between concepts (such as the relationship between medicinal materials and efficacy). Its core is to express the complex relationships within the TCM field in the form of a semantic network, used to assist in diagnosis, drug compatibility analysis, and other scenarios, supporting systematic exploration in research and application.

[0086] Intelligent agents are entities that can perceive their environment and take autonomous actions to achieve their goals, and typically possess learning, reasoning, and decision-making abilities.

[0087] A correlation matrix is ​​a two-dimensional matrix that quantifies the similarity or difference between pairs of objects in a set. Each element represents the degree of correlation between the corresponding sets. By systematically comparing objects such as shapes and point sets, it can reveal overall correlation patterns and is often used in pattern recognition or data analysis to help understand complex relationships from a global perspective.

[0088] Weight matrix: A structured tool used to quantify the importance or strength of association between data attributes, typically represented as a two-dimensional matrix. Each element reflects the weight value between different feature pairs. By adjusting these weights, the model's focus on key features can be optimized. It is commonly used in feature selection, data analysis, or model training to help uncover core association patterns between data.

[0089] The monarch, minister, assistant, and envoy principle: It is the core principle of traditional Chinese medicine (TCM) compatibility. By clarifying the hierarchical role division of drugs in a prescription (the principal drug "monarch", the supplementary drug "minister", the synergistic drug "assistant", and the harmonizing drug "envoy"), a multi-dimensional collaborative system is constructed. Its core lies in integrating different drug effects with a clear primary and secondary structure, which not only strengthens the core therapeutic effect but also balances the conflict of drug properties. At the same time, it takes into account secondary symptoms and the regulation of drug toxicity, thereby optimizing the accuracy and safety of treating complex diseases and reflecting the unique thinking of "holistic regulation" in TCM.

[0090] Pagerank: It is a static weight distribution algorithm based on graph structure, which evaluates the global importance by analyzing the link relationships between nodes. Its core idea is that "the node linked by more high-weight nodes is more important", and it is commonly used in scenarios such as web page ranking and social network influence analysis.

[0091] Dynamic adaptability: It refers to whether a method or system can adjust the recommendation strategy in real time according to the changes in the patient's condition, individual differences, and new data input.

[0092] Compound prescription: It refers to a combination of traditional Chinese medicines, such as Danggui Buxue Decoction and Liuwei Dihuang Pills.

[0093] Syndrome type: It refers to the classification of traditional Chinese medicine syndrome differentiation (disease phenotype), such as qi and blood deficiency syndrome and kidney yin deficiency syndrome.

[0094] Pathway: It refers to a biological pathway or signal network, such as the tumor necrosis factor (TNF) signaling pathway.

[0095] Existing traditional Chinese medicine recommendation methods have many deficiencies. For example, (1) the traditional Chinese medicine recommendation method based on syndrome information using graph neural network has a too high computational complexity and is difficult to be applied to scenarios such as real-time recommendation and clinical diagnosis. The interpretability of the model is insufficient. The complex structures of graph neural network and multi-layer perceptron make the recommendation results difficult to be intuitively understood and explained, which may affect the trust of doctors and patients in the recommendation results. In addition, the dynamic adaptability of the model may be insufficient, lacking the ability to adapt to the dynamically changing traditional Chinese medicine diagnosis and treatment data and unable to be updated and optimized in time to deal with new diseases or treatment methods. (2) The traditional Chinese medicine recommendation method based on multi-graph convolutional neural network has too high data dependence. This method depends on the data sets of classic medical books such as Treatise on Febrile Diseases and may not be able to fully cover the complex situations and diverse symptom combinations in modern traditional Chinese medicine diagnosis and treatment. And the model complexity is too high, resulting in a large amount of computing resources required. At the same time, it lacks the ability to adapt to the dynamically changing traditional Chinese medicine diagnosis and treatment data and cannot be updated and optimized in time to deal with new diseases or treatment methods.

[0096] To solve the problems that the existing traditional Chinese medicine recommendation methods have relatively low accuracy in traditional Chinese medicine recommendation and relatively poor adaptability to dynamically changing data.

[0097] Refer to Figure 1 This application provides a flowchart illustrating a TCM recommendation method based on multimodal fusion. This TCM recommendation method based on multimodal fusion is applied to electronic devices, such as servers or mobile terminals. Figure 1 As shown, the TCM recommendation method based on multimodal fusion may include the following steps:

[0098] Step S100: Obtain multimodal data of traditional Chinese medicine containing text data, image data, audio data, and video data;

[0099] Step S200: Preprocess the TCM multimodal data to obtain preprocessed TCM multimodal data;

[0100] Step S300: Construct a heterogeneous graph based on the core representative objects and the preprocessed TCM multimodal data, wherein the core representative objects include the core medicinal materials in the TCM multimodal data;

[0101] Step S400: Input the heterogeneous graph into the graph neural network, and use the graph neural network to fuse the preprocessed TCM multimodal data to obtain TCM multimodal fused data;

[0102] Step S500: Construct a knowledge graph based on multimodal fusion data of traditional Chinese medicine to obtain a knowledge graph of traditional Chinese medicine;

[0103] Step S600: Divide the TCM knowledge graph into subgraphs for multi-agent processing according to efficacy and safety, and obtain the medicinal material subgraph associated with the efficacy agent and the safety subgraph associated with the safety agent. The multi-agent includes two types: efficacy agent and safety agent. The efficacy agent includes the combination of medicinal materials strategy, and the safety agent includes the review of medicinal material safety strategy.

[0104] Step S700: Based on the medicinal material subgraph and the safety subgraph, the selection of medicinal materials is optimized through collaborative iteration by combining medicinal material strategies and reviewing the safety of medicinal materials, so as to obtain the optimal compound prescription recommended by traditional Chinese medicine.

[0105] Step S800: Input the optimal compound into the fine-tuned large language model to determine the target compound recommended by traditional Chinese medicine.

[0106] In this embodiment, multimodal data of Traditional Chinese Medicine (TCM) including text, image, audio, and video data is acquired and preprocessed to obtain preprocessed TCM multimodal data. A heterogeneous graph is constructed based on core representative objects and the preprocessed TCM multimodal data. The core representative objects include core medicinal materials from the TCM multimodal data. The heterogeneous graph is input into a graph neural network, which fuses the preprocessed TCM multimodal data to obtain fused TCM multimodal data. This fused data enriches the information dimensions of TCM data, possesses high flexibility and scalability, and can meet the data application needs in various scenarios. Finally, a knowledge graph is constructed based on the fused TCM multimodal data to obtain a TCM knowledge graph, achieving a visual representation of TCM data. The TCM knowledge graph is then divided into subgraphs for multi-agent processing based on efficacy and medication safety. This results in a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent. The multi-agent agents include efficacy agents and safety agents. The efficacy agent includes a medicinal material combination strategy, while the safety agent includes a medicinal material safety verification strategy. Based on the medicinal material and safety subgraphs, the selection of medicinal materials is optimized through collaborative iteration using the medicinal material combination strategy and the medicinal material safety verification strategy. This yields the optimal TCM prescription. By decomposing the TCM knowledge graph into subgraphs and corresponding agents, and establishing a dynamic collaboration mechanism between agents, accurate recommendation and safety verification of TCM prescriptions are achieved. Finally, the optimal prescription is input into a fine-tuned large language model to determine the target TCM prescription, generating a complete prescription containing precise dosage, decoction method, and treatment course. This improves the accuracy of TCM recommendations and enhances adaptability to dynamically changing data.

[0107] The above-mentioned preprocessing of TCM multimodal data to obtain preprocessed TCM multimodal data can be achieved by processing each modality of the TCM multimodal data in different ways. Processing methods can include feature extraction, normalization, etc.

[0108] The above-mentioned construction of a heterogeneous graph based on the core representative objects and preprocessed TCM multimodal data can be achieved by taking the core representative objects as entity nodes, taking the preprocessed TCM multimodal data as modal nodes, constructing edges connecting entity nodes and modal nodes, and constructing a heterogeneous graph based on the relationship between nodes and edges.

[0109] The aforementioned TCM multimodal fusion data can be obtained by using feature vectors fused through graph neural networks as TCM multimodal fusion data.

[0110] The above-mentioned method of inputting the optimal compound into a fine-tuned large language model to determine the target compound recommended by traditional Chinese medicine can be achieved by fine-tuning a pre-trained large language model using the Chinese Pharmacopoeia, and then inputting the optimal compound into the fine-tuned large language model. This can complete the optimal compound and thus determine the target compound recommended by traditional Chinese medicine. The pre-trained large language model can be obtained by training the large language model using existing datasets.

[0111] In some implementations, the multimodal data of traditional Chinese medicine is preprocessed to obtain preprocessed multimodal data of traditional Chinese medicine, including:

[0112] Word embedding is used to preprocess the text data in the multimodal data of traditional Chinese medicine to obtain the first feature vector;

[0113] A convolutional neural network is used to preprocess the image data in the multimodal data of traditional Chinese medicine to obtain the second feature vector;

[0114] A deep learning model is used to preprocess the audio data in the multimodal data of traditional Chinese medicine to obtain the third feature vector;

[0115] A convolutional neural network is used to preprocess the video data in the multimodal data of traditional Chinese medicine to obtain the fourth feature vector;

[0116] The first, second, third, and fourth eigenvectors are normalized and dimensionality reduced to obtain preprocessed TCM multimodal data.

[0117] In this embodiment, word embedding is used to preprocess the text data in the TCM multimodal data to obtain a first feature vector; a convolutional neural network is used to preprocess the image data in the TCM multimodal data to obtain a second feature vector; a deep learning model is used to preprocess the audio data in the TCM multimodal data to obtain a third feature vector; a convolutional neural network is used to preprocess the video data in the TCM multimodal data to obtain a fourth feature vector; the first, second, third, and fourth feature vectors are then normalized and dimensionality reduced to obtain the preprocessed TCM multimodal data. Thus, using different methods to process different modalities of data yields more accurate feature data, reduces the amount of data processing required for subsequent multimodal data fusion, improves data processing efficiency, and enriches the information dimensions of TCM data through extraction and representation.

[0118] In some implementations, a knowledge graph is constructed based on multimodal fusion data of traditional Chinese medicine (TCM) to obtain a TCM knowledge graph, including:

[0119] Core objects in the multimodal data of traditional Chinese medicine are selected as entity nodes. The core objects include medicinal materials and symptoms in the multimodal data of traditional Chinese medicine.

[0120] Using multimodal fusion data of traditional Chinese medicine as attribute information for entity nodes;

[0121] Construct relationship edges based on the relationships between entity nodes, and preset initial weights for the relationship edges;

[0122] Construct an initial knowledge graph based on entity nodes, attribute information, and relation edges;

[0123] Markov decision is used to adjust the initial weights of relation edges in the initial knowledge graph to obtain optimized weights;

[0124] The optimized weights are used to replace the initial weights of the relation edges in the initial knowledge graph to construct a knowledge graph for traditional Chinese medicine.

[0125] In this embodiment, core objects from TCM multimodal data are selected as entity nodes, including medicinal materials and symptoms. The fused TCM multimodal data is used as attribute information for these entity nodes. Relationship edges are constructed based on the relationships between entity nodes, and initial weights are preset for each edge. An initial knowledge graph is constructed based on the entity nodes, attribute information, and relationship edges. Markov decision-making is used to adjust the initial weights of the relationship edges in the initial knowledge graph, resulting in optimized weights. These optimized weights replace the initial weights of the relationship edges in the initial knowledge graph, thus constructing the TCM knowledge graph. In this way, the knowledge graph enables the visual representation of TCM data, and the dynamic decision-making capability of reinforcement learning (i.e., Markov decision-making) optimizes the weight allocation and path reasoning efficiency of entity relationships within the knowledge graph.

[0126] In some implementations, the TCM knowledge graph is divided into subgraphs for multi-agent processing according to efficacy and medication safety, resulting in a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent, including:

[0127] The knowledge graph of traditional Chinese medicine is constructed as a main graph structure containing nodes, binary edges, hyperedges, and node feature vectors.

[0128] By learning the node feature vectors in the main graph structure through multiple agents, and using the feature projection matrix to enhance the node feature vectors during the learning process, the enhanced node feature vectors corresponding to each agent are obtained.

[0129] Based on the augmented node feature vector corresponding to each agent, the similarity between nodes is calculated, and a sparse adjacency matrix corresponding to each agent is constructed based on the similarity.

[0130] Construct the degree matrix based on the sparse adjacency matrix corresponding to each agent;

[0131] Subtracting the degree matrix from the sparse adjacency matrix yields the nonnormalized Laplace matrix;

[0132] Clustering is performed on the eigenvectors in the non-normalized Laplacian matrix to obtain multiple clustering results;

[0133] Each clustering result is processed as a subgraph by an agent to obtain a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent.

[0134] In this embodiment, the TCM knowledge graph is constructed as a main graph structure containing nodes, binary edges, hyperedges, and node feature vectors. Multiple agents learn the node feature vectors in the main graph structure, and feature projection matrices are used to enhance the node feature vectors during the learning process, resulting in enhanced node feature vectors for each agent. Based on the enhanced node feature vectors for each agent, the similarity between nodes is calculated, and a sparse adjacency matrix is ​​constructed for each agent based on the similarity. A degree matrix is ​​constructed based on the sparse adjacency matrix for each agent. The degree matrix and the sparse adjacency matrix are subtracted to obtain a non-normalized Laplacian matrix. The feature vectors in the non-normalized Laplacian matrix are clustered to obtain multiple clustering results. Each clustering result is processed as a subgraph by an agent to obtain a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent. In this way, multi-dimensional structural cognition is achieved. The safety agent strengthens the similarity of taboo features to construct a taboo subgraph with high similarity connections in the projection space; the efficacy agent enhances the similarity of meridian tropism features to form dense connections between nodes with similar meridian tropism. The differentiation mechanism enables each agent to accurately capture task-related topological patterns in its own feature space, laying a solid data foundation for subsequent TCM recommendations.

[0135] In some implementations, based on the herb subgraph and the safety subgraph, the selection of herbs is optimized through collaborative iterative optimization using a combination of herb selection strategies and a herb safety review strategy, resulting in the optimal compound prescription recommended by traditional Chinese medicine, including:

[0136] A combination of medicinal materials strategy was adopted, in which medicinal materials were selected from the medicinal material sub-map and combined to obtain the first compound prescription;

[0137] Based on the safety subgraph, the safety of medicinal materials in the first compound formula is reviewed using a medicinal material safety review strategy to obtain the second compound formula.

[0138] The strategy of combining medicinal materials and the strategy of reviewing the safety of medicinal materials are synergistically iteratively optimized until the second compound is the same in two consecutive synergistic iterative optimizations, thus obtaining the optimal compound recommended by traditional Chinese medicine.

[0139] In this embodiment, a combination of medicinal materials strategy is adopted to select and combine medicinal materials from the medicinal material subgraph to obtain a first compound prescription. Based on the safety subgraph, a medicinal material safety audit strategy is used to audit the medicinal materials in the first compound prescription to obtain a second compound prescription. The combination of medicinal materials strategy and the medicinal material safety audit strategy are then iteratively optimized in a coordinated manner until the second compound prescription is the same in two consecutive iterative optimizations, thus obtaining the optimal compound prescription recommended by traditional Chinese medicine. In this way, the coordinated iterative optimization of the combination of medicinal materials strategy and the medicinal material safety audit strategy, i.e., the introduction of a two-layer divide-and-conquer architecture auxiliary retrieval scheme, not only significantly reduces the spatiotemporal scope of graph traversal retrieval and significantly reduces the computational resource requirements of traditional traditional Chinese medicine recommendation schemes, but also improves the accuracy of traditional Chinese medicine recommendations.

[0140] In some implementations, the herbal sub-graph includes the relationship between symptoms and the therapeutic effects of the herbs. A combination herbal strategy is employed, selecting herbs from the herbal sub-graph for combination to obtain a first compound prescription, including:

[0141] Based on the patient's symptoms, calculate the overall efficacy score of each herb in the herb sub-chart for the symptoms;

[0142] The combination of medicinal materials is iteratively adjusted, including: selecting a preset number of medicinal materials according to the comprehensive efficacy score from high to low to obtain the initial compound prescription;

[0143] Medicinal herbs with a comprehensive efficacy score greater than or equal to the first preset value in the initial compound prescription will be retained;

[0144] After removing herbs with a comprehensive efficacy score lower than the second preset value from the initial compound prescription, the herbs are replenished to the preset number according to the comprehensive efficacy score from high to low.

[0145] The first compound is obtained when the changes in the medicinal components in the compound are less than the third preset value after two consecutive rounds of iterative adjustments of the combined medicinal material strategy, or when the fluctuation of the total efficacy score of all medicinal materials in the compound is less than the fourth preset value.

[0146] In this embodiment, based on the patient's symptoms, the comprehensive efficacy score of each herb in the herbal sub-graph is calculated. Iterative adjustments are then made according to a herbal combination strategy, including: selecting a preset number of herbs from high to low comprehensive efficacy scores to obtain an initial compound; retaining herbs in the initial compound with comprehensive efficacy scores greater than or equal to a first preset value; removing herbs in the initial compound with comprehensive efficacy scores less than a second preset value, and replenishing the herbs to a preset number from high to low comprehensive efficacy scores; until the changes in herbal components in the compound after two consecutive rounds of iterative adjustments by the herbal combination strategy are less than a third preset value, or the fluctuation in the total efficacy score of all herbs in the compound is less than a fourth preset value, to obtain the first compound. Thus, by using comprehensive efficacy scores to screen herbs related to the patient's symptoms, the principles of monarch, minister, assistant, and guide in traditional Chinese medicine compound prescriptions are met, resulting in a more effective compound and improving the accuracy of traditional Chinese medicine recommendations.

[0147] The values ​​of the first, second, third, and fourth preset values ​​mentioned above can be changed according to actual circumstances, and this embodiment does not impose specific limitations.

[0148] In some implementations, the safety subgraph includes medicinal herb entity nodes and incompatibility relationship edges. Based on the safety subgraph, a medicinal herb safety audit strategy is used to audit the medicinal herbs in the first compound formula to obtain the second compound formula, which includes:

[0149] The safety strategy for medicinal materials is adjusted iteratively, including: obtaining the single-drug toxicity of each medicinal material and removing medicinal materials in the first compound with single-drug toxicity greater than or equal to the first safety threshold.

[0150] Calculate the total toxicity of all medicinal materials in the first compound formula, and calculate the contribution of each medicinal material to the therapeutic effect. Based on the contribution of each medicinal material to the therapeutic effect and the toxicity of each single drug, calculate the toxicity efficiency ratio of each medicinal material.

[0151] If the total toxicity exceeds the second safety threshold, remove the medicinal materials with a low toxicity-efficiency ratio.

[0152] If there is a compatibility incompatibility relationship edge in the first compound prescription, then the medicinal material pair is obtained based on the two medicinal material entity nodes corresponding to the compatibility incompatibility relationship edge;

[0153] Calculate the topological fit of each herb in the herb pair, and remove herbs with low topological fit from the herb pair;

[0154] The second compound formula is obtained when the medicinal materials in the compound formula remain unchanged after two consecutive rounds of iterative adjustments to the medicinal material safety strategy.

[0155] In this embodiment, iterative adjustments are made based on the safety strategy for medicinal materials, including: obtaining the single-drug toxicity of each medicinal material; removing medicinal materials in the first compound with a single-drug toxicity greater than or equal to a first safety threshold; calculating the total toxicity of all medicinal materials in the first compound, and calculating the contribution of each medicinal material to the therapeutic effect, and calculating the toxicity-efficiency ratio of each medicinal material based on the contribution of the therapeutic effect and the single-drug toxicity; if the total toxicity is greater than a second safety threshold, removing medicinal materials with a low toxicity-efficiency ratio; if there is a contraindication relationship edge in the first compound, obtaining a medicinal material pair based on the two medicinal material entity nodes corresponding to the contraindication relationship edge; calculating the topological fit of each medicinal material in the medicinal material pair, and removing medicinal materials with low topological fit in the medicinal material pair; until the medicinal materials in the compound remain unchanged after two consecutive rounds of iterative adjustments based on the safety strategy for medicinal materials, a second compound is obtained. Thus, by calculating the conflicts between medicinal materials based on the contraindications between them and their own toxicity attributes, and removing conflicting medicinal materials, a safe compound is finally obtained, thereby improving the accuracy of traditional Chinese medicine recommendations.

[0156] The values ​​of the first and second safety thresholds mentioned above can be changed according to actual conditions, and this embodiment does not impose specific limitations.

[0157] The removal of herbs with low toxicity efficiency can be done by removing only the herb with the lowest toxicity efficiency, or by removing multiple herbs with low toxicity efficiency. The specific number of herbs to be removed can be changed according to the actual situation, and this embodiment does not impose a specific limitation.

[0158] The removal of herbs with low topological fit in the above-mentioned herbal medicine pair can be done by removing only the herbal medicine with the lowest topological fit, or by removing multiple herbs with low topological fit. The specific number of herbs to be removed can be changed according to the actual situation, and this embodiment does not make a specific limitation.

[0159] To facilitate understanding by those skilled in the art, a set of preferred embodiments is provided below:

[0160] This embodiment proposes an auxiliary retrieval scheme based on multimodal TCM data fusion using graph neural networks, knowledge graph representation using reinforcement learning, and a two-layer divide-and-conquer architecture. The scheme preprocesses TCM data from different modalities, mapping them to a unified dimension and semantic space. Then, based on graph neural networks, it fuses the preprocessed multimodal data to obtain highly representative core TCM data information, extracting and integrating key information and data from massive TCM multimodal data. Based on the core objects and data information in the corresponding TCM data, a complex TCM knowledge graph is constructed, allowing for dynamic and visual adjustments. The weights of edges between entities are dynamically adjusted using reinforcement learning strategies. Finally, combined with a two-layer divide-and-conquer architecture and a large language model, accurate and efficient drug diagnosis and treatment assistance schemes are provided. Compared to traditional methods, this embodiment effectively integrates massive TCM data and explores its potential through multimodal fusion, and constructs a complex yet detailed TCM knowledge graph using reinforcement learning strategies. By combining a two-layer divide-and-conquer architecture with a large language model, efficient and accurate recommendations for traditional Chinese medicine were achieved. This method also overcomes the shortcomings of traditional graph neural network methods, such as their difficulty in intuitive understanding and potential lack of dynamic adaptability. (See reference...) Figure 2 The technical solution of this embodiment specifically includes the following contents:

[0161] I. A Multimodal Traditional Chinese Medicine Data Fusion and Representation Scheme Based on Graph Neural Networks.

[0162] To effectively extract key information from traditional Chinese medicine (TCM) data, this embodiment proposes a multimodal TCM data fusion and representation scheme based on graph neural networks. This scheme utilizes graph neural networks for multimodal data fusion, achieving the extraction and representation of multimodal TCM data. Its core lies in using graph neural networks combined with multimodal fusion technology to transform unstructured multimodal TCM data into structured data, enriching the information dimensions of TCM data and deeply exploring its potential value. This scheme not only extracts core information from massive amounts of multimodal TCM data but also possesses high flexibility and scalability, meeting the data application needs of various scenarios.

[0163] 1. Introduction to multimodal data preprocessing.

[0164] Multimodal data preprocessing is the first step in this embodiment. Its core idea is to preprocess massive amounts of multimodal data, including TCM texts, images, and audio, to reduce data redundancy and retain valuable content as much as possible. Simultaneously, it effectively reduces the amount of data processing required for subsequent multimodal data fusion using graph neural networks (GNNs), improving data processing efficiency. (Refer to...) Figure 3 The specific process is as follows:

[0165] (1) Data input: Receive multi-modal data of traditional Chinese medicine, including text data such as classical literature, medical records, treatises, etc., traditional Chinese medicine image data, audio data, and video data.

[0166] (2) Data preprocessing: Preprocess data of different modalities separately, and map the features extracted from different modalities to the same semantic space. The specific preprocessing methods are as follows:

[0167] 1) Text data: Use text cleaning technology to remove special symbols such as "@", "#" in the text content and filter out some stop words such as "不如 (not as good as)", "与其 (compared with)", etc., to standardize the text data. Then, use word segmentation technology and natural language processing technology (NLP) to extract and classify the core words in the text data. Finally, use word embedding technology to convert the processed text data into a numerical feature vector representation (i.e., the first feature vector).

[0168] 2) Image data: Normalize the size of the image data, uniformly adjust it to the same size, and standardize the color to meet the requirements of model input. Then, use a convolutional neural network (CNN) model to extract the feature vector of the image data (i.e., the second feature vector).

[0169] 3) Audio data: Unify the sampling rate of the audio data to a fixed value to ensure the consistency of the sampling rates of different audio files. Then, perform noise reduction processing on the audio data to effectively remove background noise and make the audio content clear. Finally, use the pre-trained deep learning model OpenL3 to extract audio feature vectors of a fixed length (i.e., the third feature vector).

[0170] 4) Video data: Decode the video into a series of consecutive video frames, and then extract key frames according to the method of sampling at intervals. The key frame feature vectors are also extracted using a convolutional neural network (CNN) model, and the semantic feature vectors of the key frames are extracted respectively as the video feature vector (i.e., the fourth feature vector).

[0171] (3) L2 normalization processing: Perform L2 normalization processing on the feature vectors extracted from each modality to eliminate the scale difference between the features of different modalities. The formula is:

[0172] ;

[0173] where, represents the feature vector after L2 normalization processing, represents the elements in the feature vector.

[0174] (4) Dimensionality reduction: Perform principal component analysis (PCA) linear dimensionality reduction on each individual mode to unify the features of different modes to the same dimension, which facilitates subsequent processing.

[0175] (5) Data transfer: The processed feature vector (i.e., the preprocessed TCM multimodal data) is transferred to the TCM multimodal data fusion module.

[0176] 2. Introduction to multimodal data fusion in traditional Chinese medicine.

[0177] Reference Figure 4 This is a flowchart illustrating the process of the TCM multimodal data fusion module. The core method of TCM multimodal data fusion is to use a Generative Neural Network (GNN) to fuse the feature vectors of each modality extracted from the multimodal data preprocessing module, providing relevant information for the subsequent construction of a TCM knowledge graph.

[0178] 2.1. Construction of heterogeneous graphs.

[0179] The construction of heterogeneous graphs is the first step in the TCM multimodal data fusion scheme. It primarily organizes each TCM entity and its associated multimodal data (images, text, and audio data, etc.) into a graph structure, providing the foundation for subsequent feature aggregation and multimodal data fusion based on graph neural networks (GNNs). The specific process is as follows:

[0180] (1) The nodes of the graph are defined as follows, and a unique index is assigned to each entity node:

[0181] 1) Entity node definition: The core representative object (e.g., Ganoderma lucidum, deer antler and other core medicinal materials) is defined as an entity node.

[0182] 2) Modal node definition: The feature vector of each modality is used as the corresponding modal node.

[0183] (2) Definition of graph edge: connects entity nodes and corresponding modal nodes.

[0184] (3) Construction of adjacency relationships: Based on the relationship between nodes and edges, define and construct the corresponding adjacency matrix for each edge. The rows and columns of the matrix represent different types of nodes, with a value of 1 indicating a connection between nodes and a value of 0 indicating no connection between nodes, thus providing a foundation for subsequent multimodal data fusion using graph neural networks.

[0185] 2.2. GNN enables the fusion of multimodal data.

[0186] This is the second step in the TCM multimodal data fusion solution, primarily responsible for utilizing graph neural networks (GNNs) to achieve multimodal data fusion. GNNs effectively capture the complex relationships between nodes and edges in a graph structure, thereby achieving deep feature fusion. Specific process:

[0187] (1) Definition of message passing rules for heterogeneous graphs.

[0188] Forward propagation: Modal nodes are propagated to entity nodes to form a multimodal joint representation.

[0189] (2) Feature initialization and aggregation.

[0190] 1) Feature initialization: The feature vector of each node (including entity nodes and modal nodes) in the heterogeneous graph is used as the initial input. The feature vector of an entity node can be its basic attributes (such as name, category, etc.), while the feature vector of a modal node is the feature vector extracted from multimodal data (such as images, text, and audio).

[0191] 2) Feature aggregation: Through the aggregation mechanism of GNN, the feature vectors of adjacent nodes are weighted and summed or concatenated to generate new feature representations. This process can capture the correlation information between nodes, providing a foundation for subsequent feature fusion.

[0192] (3) Construction of graph neural networks.

[0193] Graph Convolutional Layer (GCN): This layer updates node features using graph convolutional layers. Each node's feature vector interacts with the features of its neighboring nodes through an adjacency matrix, thereby capturing local structural information. The formula is as follows:

[0194] ;

[0195] in, This indicates adding a self-connected adjacency matrix. Degree matrix, The graph neural network represents the first... The node feature matrix of the layer Represents the weight matrix. This represents the activation function. The graph neural network represents the first... The node feature matrix of the layer.

[0196] (4) Information transmission and integration.

[0197] 1) Information Transmission: Through the propagation mechanism of GNN, the feature information of modal nodes is transmitted to entity nodes. The adjacency matrix records the connection relationships between nodes, and GNN uses this structure to realize the layer-by-layer transmission of features.

[0198] 2) Feature Fusion: At the entity node level, feature vectors from different modalities are fused. Fusion methods can include simple concatenation, weighted summation, or generating a unified feature representation through nonlinear transformation, as shown in the following formula:

[0199] ;

[0200] in, It is a physical node The final fusion features (i.e., multimodal fusion data of traditional Chinese medicine). It is a physical node In the Features of the layer It is the first The modal node at the ... Features of the layer It is a weight matrix. It is an activation function.

[0201] 3) Reasoning logic: Information transmission ensures that the features of modal nodes can be passed to entity nodes layer by layer, while feature fusion generates a unified feature representation through different fusion methods (i.e., splicing, weighted summation, and nonlinear transformation), thereby achieving deep fusion of multimodal data.

[0202] (5) Output and optimization of fusion results.

[0203] 1) Feature representation output: The final generated feature vector (i.e. TCM multimodal fusion data) serves as the fusion feature of entity nodes, which can simultaneously reflect the information of multimodal data.

[0204] 2) Loss function:

[0205] ;

[0206] in, Indicates the number of prediction results. This indicates that it is a genuine label. This represents the predicted label (i.e., the value predicted by the model).

[0207] 3) Model optimization: Update the model parameters through backpropagation and optimization algorithms (such as the Adam optimization algorithm) to minimize the value of the loss function, thereby optimizing the fusion effect.

[0208] Through the above process, this embodiment preprocesses multimodal TCM data using methods such as data preprocessing, normalization, and dimensionality reduction. Based on graph neural networks, data from different modalities are fused. This graph neural network-based multimodal TCM data fusion and representation scheme not only uncovers the potential key information in TCM data but also significantly reduces the redundancy of the data.

[0209] II. Knowledge Graph Representation Scheme Based on Reinforcement Learning.

[0210] Reference Figure 5This is a schematic diagram illustrating the knowledge graph construction process based on reinforcement learning. To effectively represent traditional Chinese medicine (TCM) data, this embodiment proposes a knowledge graph representation scheme based on reinforcement learning. This scheme utilizes knowledge graphs to achieve visual representation of TCM data and leverages the dynamic decision-making capabilities of reinforcement learning to optimize the weight allocation of entity relationships and the efficiency of path reasoning in the knowledge graph.

[0211] 1. Introduction to the construction of a knowledge graph for traditional Chinese medicine.

[0212] The first step in this solution is constructing a Traditional Chinese Medicine (TCM) knowledge graph (i.e., the initial knowledge graph). Its core idea is to transform TCM data into a visualized graph structure. Specific process:

[0213] (1) Data reception: Receive TCM multimodal fusion data transmitted to this module by the multimodal data fusion module.

[0214] (2) Definition of entity nodes: Core objects in TCM multimodal data, such as Angelica sinensis and diarrhea, are defined as entity nodes.

[0215] (3) Definition of entity node attributes: The multimodal fusion data of traditional Chinese medicine obtained from the data reception will be used as the attribute information of the entity node.

[0216] (4) Definition of relation edges: Based on the type of entity node and the relationship between entity nodes, such as treatment relationship (Chinese medicine → disease), compatibility relationship (Chinese medicine → Chinese medicine), and contraindication relationship (Chinese medicine < → Chinese medicine), etc., relation edges are assigned initial weights. The initial weights are preset values.

[0217] 2. Introduction to dynamic weight adjustment based on Markov decision-making.

[0218] The second step of this scheme is dynamic weight adjustment based on Markov decision-making. Its core objective is to dynamically optimize the weight allocation of entity relationships (such as "treatment," "compatibility," and "contraindications") in the TCM knowledge graph (i.e., the initial knowledge graph) through reinforcement learning strategies. This aims to improve the practicality and safety of recommended TCM pathways, providing an information and data foundation for the subsequent two-layered treatment architecture. Specific process:

[0219] (1) Markov state definition: The combination of multiple entity nodes is a composite state, reflecting the relationship between drug pairs, prescriptions or other multi-dimensional relationships.

[0220] (2) Definition of state transition rules:

[0221] 1) Definition of transition probability calculation: From the composite state Transition to composite state probability The formula is:

[0222] ;

[0223] 2) Definition of transfer constraint rule: When the entity node to be transferred has a taboo relationship, the transfer probability is forced to 0.

[0224] (3) Dynamic weight adjustment mechanism:

[0225] 1) Positive Feedback Rule: When a path is verified to be valid, the weight of the relevant edges is increased, and the formula is as follows:

[0226] ;

[0227] in, This represents the initial weight value. Indicates the learning rate. This indicates the therapeutic effect score.

[0228] 2) Reverse Feedback Rule: If a path triggers a taboo relationship or becomes invalid, the weight of the relevant edges is reduced. The formula is as follows:

[0229] ;

[0230] in, This represents the initial weight value. Indicates the penalty coefficient. Indicates the severity of the violation (e.g., 0.95 indicates a serious violation).

[0231] Through the above process, this embodiment obtains a TCM knowledge graph that has been dynamically adjusted by Markov decision, including entity nodes, the relationships between edges, and their corresponding weights.

[0232] III. An auxiliary retrieval scheme based on a two-layer divide-and-conquer architecture.

[0233] To address the challenges of existing methods in uniformly processing knowledge graphs containing multi-dimensional information such as contraindications, drug interactions, and patient symptoms, and the response delays caused by full-graph traversal analysis of ultra-large-scale knowledge graphs, which impact real-time clinical decision-making, this embodiment proposes a two-layer divide-and-conquer architecture to achieve intelligent processing of complex knowledge graphs, referring to... Figure 6 This scheme subdivides the original knowledge graph through a two-layer architecture of graph deconstruction and agent interaction, thereby effectively processing and utilizing conflicting information between graphs, and transmitting the processed information to a large language model to assist in diagnosis.

[0234] In this embodiment, the main graph of the Traditional Chinese Medicine knowledge graph is redefined as follows:

[0235] Main graph structure: Defined as a weighted hypergraph G=(V, E, H), where the node set V includes medicinal materials, syndrome types, and meridian tropism, etc.; the binary edges E include incompatibilities and synergistic effects, etc.; and the hyperedges H include multi-node relationships connecting compound prescriptions, syndrome types, and pathways. Node characteristics are: each node... There are eigenvectors , Represents the dimension of the feature vector. It includes properties such as the four natures and five flavors, meridian tropism, and toxicity.

[0236] 1. Introduction to the diagram deconstruction layer.

[0237] The graph deconstruction layer is primarily used to cut a complex main graph into subgraphs suitable for different agents. Each subgraph contains key information required for a specific task and is passed to the subsequent agent interaction layer. The specific process is as follows:

[0238] (1) Dynamic Feature Enhancement: This method enhances the target agent's features by employing a learnable feature projection matrix, mapping the original features to a task-adaptive semantic space through linear transformation. This mechanism enhances the agent's ability to perceive key attribute information through differentiated feature representation, thereby optimizing subgraph segmentation based on feature correlation. Its mathematical expression is as follows:

[0239] ;

[0240] in, The node feature vector represents the inherent properties of medicinal materials, such as their four natures, five flavors, and meridian tropism. Represents intelligent agents A dedicated trainable weight matrix to achieve Via 3D feature space transformation; This represents the feature offset vector, used to adjust the feature distribution after projection; Indicates intelligent agent The optimized feature vector (i.e., the enhanced node feature vector) enhances the weights of task-related features.

[0241] The technical solution of this embodiment establishes a connection between different subgraphs in the main graph and their corresponding agents, effectively reducing redundant calculations and processing.

[0242] (2) Similarity calculation and graph reconstruction.

[0243] This embodiment is based on an intelligent agent. The corresponding enhanced node feature vectors are used to construct a dynamic graph structure, and task-aware neighborhood discovery is achieved through attention-driven similarity metrics. The computational process is as follows:

[0244] In intelligent agents In the feature space, cosine similarity is used to measure the feature association strength between nodes. :

[0245] ;

[0246] in, Indicates the first 1 node Indicates the first 1 node Indicates intelligent agent After optimization The feature vector of each node Indicates intelligent agent After optimization The feature vector of each node This represents the function for calculating cosine similarity.

[0247] Then, a sparse adjacency matrix is ​​generated based on the similarity threshold to achieve adaptive reconstruction of the topology:

[0248] ;

[0249] in, Represents a sparse adjacency matrix. This represents the similarity threshold. Different agents correspond to different similarity thresholds. The similarity threshold can be set in advance by setting a similarity threshold range or by taking the top 10% of similarity values ​​from a sparse adjacency matrix as the similarity threshold.

[0250] This embodiment achieves multi-dimensional structural cognition. The safety agent enhances the similarity of taboo features to construct a taboo subgraph with high similarity connections in the projection space. The pharmacological agent enhances the similarity of meridian features, forming dense connections between nodes with similar meridian tropism. This differentiation mechanism enables each agent to accurately capture task-related topological patterns in its dedicated feature space.

[0251] (3) Spectral clustering-driven multi-agent subgraph segmentation.

[0252] After completing the adjacency relationship modeling of the multi-agent system, this embodiment uses a spectral clustering algorithm to achieve subgraph segmentation. The specific implementation steps are as follows: First, an adjacency matrix is ​​constructed based on the interaction topology of each agent. , where matrix elements Characterizing intelligent agents With the The node and the first The connection strength between nodes is then determined. Next, a degree matrix D is constructed, which is a diagonal matrix whose diagonal elements are defined as follows:

[0253] ;

[0254] in, Indicates the total number of nodes. For intelligent agents The corresponding topology network The sum of the connectivity of each node. Then, the core analytical tool in graph theory, the nonnormalized Laplacian matrix, is constructed, with the expression:

[0255] ;

[0256] in, Represents the non-normalized Laplace matrix. Degree matrix, This represents a sparse adjacency matrix.

[0257] In the feature analysis phase, the core steps are as follows: First, a special matrix (i.e., a non-normalized Laplacian matrix) is generated based on the main graph structure, from which multiple core feature vectors are extracted. These core feature vectors, when combined, best reflect the network's segmentation characteristics. Next, the K-means algorithm is used to group these core feature vector data, with each group corresponding to an agent's managed region. Finally, the internal connectivity of each agent's managed sub-network (i.e., managed region) completely maintains the original structure. This method can intelligently identify close clusters in the network, ensuring high connectivity within each divided sub-region while maintaining maximum separation between different sub-regions.

[0258] 2. Introduction to the intelligent agent interaction layer.

[0259] This embodiment employs a graph-structure-driven multi-agent collaborative architecture. By decomposing the main graph (i.e., the TCM knowledge graph) into subgraphs corresponding to agents, and establishing a dynamic collaboration mechanism between agents, it achieves accurate recommendation and safety verification of TCM compound prescriptions. The architecture specifically comprises the following three parts:

[0260] (1) The combination strategy of the pharmacodynamic agent is as follows: This agent acts on the pharmacodynamic subgraph containing the relationship between symptoms and the therapeutic relationship of medicinal materials. By calculating the correlation degree of medicinal materials related to the patient's symptoms in the pharmacodynamic subgraph, and using the agent to monitor and adjust the number of principal, assistant, adjuvant and guide drugs in real time, the agent satisfies the principal, assistant, adjuvant and guide rules of traditional Chinese medicine compound prescriptions. The specific steps are as follows:

[0261] First, based on the patient's symptoms, this intelligent agent calculates the comprehensive efficacy score (relevance) of each medicinal material from the knowledge subgraph. (i.e., the average therapeutic intensity of the medicinal material on all symptoms of the patient), that is:

[0262] ;

[0263] in, This indicates the overall therapeutic effect score. This represents a collection of patient symptoms. It refers to the symptoms of one disease in a set of patient symptoms. This refers to the medicinal materials used for symptom S. This represents the weight of the treatment edge composed of symptoms and medicinal materials. This weight is a preset value (i.e., a value pre-set based on experience).

[0264] Next, in multiple rounds of iterative adjustments (up to 10 rounds), the initial compound prescription in the initial iteration included 8 herbs. Based on the comprehensive efficacy score, priority was given to ensuring that the compound prescription contained at least 2 highly effective core herbs (highly effective core herbs are those with a comprehensive efficacy score greater than or equal to 0.8 (i.e., the first preset value)). For example, "Astragalus" would be selected first if it had a strong effect on the patient's Qi deficiency symptoms.

[0265] Then, ineffective medicinal materials (i.e., those with a comprehensive efficacy score less than 0.5 (i.e., the second preset value) are eliminated. For example, licorice is replaced if it is not effective for the current symptoms. The formula is then supplemented to eight herbs (due to the limitation of not having too many herbs in a traditional Chinese medicine compound, this embodiment includes eight herbs). Auxiliary herbs are added according to their scores from high to low; for example, Poria cocos might be selected because it has both diuretic and spleen-strengthening effects. Finally, if the changes in the formula composition are less than 10% for two consecutive rounds (i.e., the third preset value), or if the fluctuation in the total efficacy score of all herbs in the compound is less than 5% (i.e., the fourth preset value), the scheme is considered stable, optimization stops, and the first compound is obtained. The total efficacy score is the sum of the comprehensive efficacy scores of all herbs in the compound. The thresholds here can be dynamically adjusted; currently, the setting of high or medium-low thresholds is considered under normal circumstances.

[0266] It should be noted that the compound formula in this embodiment may contain only 8 herbs, but is not limited to only 8 herbs. The number of herbs in the compound formula can be modified according to the actual situation. This embodiment does not make specific limitations.

[0267] (2) The safety strategy of the security agent for reviewing medicinal materials is as follows: The agent performs a safety check on the first compound, calculates the conflict between the medicinal materials based on the incompatibilities between the medicinal materials and the toxicity of the medicinal materials themselves, and removes the conflicting medicinal materials to finally obtain a safe compound. The specific steps are as follows:

[0268] This agent operates on the security subgraph deconstructed from the Traditional Chinese Medicine knowledge graph. This subgraph contains a set of medicinal herb entity nodes. and the set of incompatible relationships (Such as the "Eighteen Incompatibilities" and other classic contraindication rules). This embodiment achieves compound optimization through a dual safety verification mechanism. Its core process is as follows: first, toxicity intensity is controlled; second, incompatibility conflicts are resolved; and finally, a second compound that meets the safety standards of the Chinese Pharmacopoeia is output.

[0269] Specifically, in stage 1, a two-level filtering is implemented through toxicity attribute calculation: (1) When the toxicity of a single drug (the toxicity of each medicinal material is recorded in the Chinese Pharmacopoeia) is greater than or equal to the first safety threshold, the highly toxic medicinal material (such as raw aconite, strychnine, etc.) is directly removed; (2) The total toxicity and toxicity efficiency ratio are calculated. If the total toxicity is greater than the second safety threshold, the inefficient and highly toxic medicinal materials are removed according to the toxicity efficiency ratio. It can be the medicinal material with the lowest toxicity efficiency ratio that is removed, or multiple medicinal materials with toxicity efficiency ratios lower than the preset value that are removed. The specific number of medicinal materials removed can be changed according to the actual situation. This embodiment does not make a specific limitation. The formula for calculating the toxicity efficiency ratio is:

[0270] ;

[0271] in, Indicates the toxicity efficiency ratio. This indicates the contribution of medicinal materials to the therapeutic effect (calculated using the PageRank value of the medicinal material on the pharmacological efficacy subgraph). Indicates total toxicity.

[0272] Phase 2 involves conflict detection on the compatibility subgraph, identifying drug pairs with edges that have incompatible relationships. Calculate the topology fit (topology fit reflects the compatibility of the medicinal material with the current compound prescription):

[0273] ;

[0274] in, Indicates topology fit. This represents the set of its neighboring nodes in the pharmacokinetics subgraph. This indicates a compliant compound.

[0275] Medicinal materials with low compatibility were removed. A dynamic iterative architecture was adopted, and the process terminated when the compound formula remained unchanged after two consecutive iterations, resulting in a second compound formula. This embodiment ensures sufficient optimization while avoiding infinite loops.

[0276] (3) Multi-agent collaboration: This embodiment proposes a knowledge graph-based dual-agent iterative optimization framework for generating traditional Chinese medicine prescriptions that balance efficacy and safety. Its core mechanism achieves dynamic optimization through the alternating execution of combined generation and safety verification, as detailed below:

[0277] This embodiment achieves dynamic optimization of the compound prescription through multi-agent iterative collaboration: First, the pharmacodynamic agent optimizes the initial compound prescription based on the patient symptom set, the herb subgraph, and the safety subgraph to generate the first compound prescription. Its core is to select the compatibility of herbs according to traditional Chinese medicine theories (such as monarch, minister, assistant, guide, nature, flavor, and meridian tropism). Subsequently, the safety agent performs contraindication verification and conflict resolution on the first compound prescription, uses drug interaction and toxicity rules in the graph to eliminate high-risk combinations, and outputs the second compound prescription. If the second compound prescription does not change in two consecutive iterations, it is considered to have reached a stable state, the loop is terminated, and the final compound prescription (i.e., the optimal compound prescription) is output.

[0278] 3. Introduction to Large Language Model-Assisted Diagnosis and Treatment.

[0279] The large-scale language model, based on the "principal, assistant, adjuvant, and guide" drug pairings selected through a divide-and-conquer architecture, combines traditional Chinese medicine knowledge graphs to analyze the properties, flavors, meridian tropism, and synergistic patterns of medicinal materials. Using the "principal drug's efficacy" as a benchmark, dosages are dynamically allocated according to the "principal and assistant" hierarchy, with fine-tuning applied to the adjuvant and guide drugs. The large-scale language model is further refined using multiple ancient Chinese medicine texts, including the *Chinese Pharmacopoeia*, to acquire domain-specific professional knowledge. The final compound prescription is input into the refined large-scale language model, which then generates a complete compound prescription (i.e., the target compound prescription) containing precise dosages, decoction methods, and treatment courses, ensuring that the target compound prescription adheres to the principle of "orderly primary and secondary components, and appropriate tonification and purgation."

[0280] Compared with the prior art, the technical solution of this embodiment has the following advantages:

[0281] This embodiment's technical solution is based on multimodal TCM data fusion using graph neural networks, knowledge graph representation using reinforcement learning, and a two-layer divide-and-conquer architecture-based auxiliary retrieval scheme. By employing multimodal data fusion, dynamic adjustment of the TCM knowledge graph using reinforcement learning, auxiliary retrieval using a two-layer divide-and-conquer architecture, and generation of target compound prescriptions using a large language model, it successfully solves the problems of high computational cost, high complexity, and difficulty in clinical application associated with traditional methods. This embodiment's technical solution not only improves the accuracy and efficiency of compound prescription recommendations and significantly reduces reliance on large amounts of computational resources, but also successfully uncovers the potential information in TCM data. Furthermore, the representational structure of the knowledge graph improves the model's interpretability, making the TCM data content more transparent and intuitive. This embodiment's technical solution is easy to understand and analyze, and has broad application prospects.

[0282] Reference Figure 7 This application also provides a TCM recommendation system based on multimodal fusion. The system includes a data acquisition unit 100, a data processing unit 200, a heterogeneous graph construction unit 300, a data fusion unit 400, a graph construction unit 500, a graph segmentation unit 600, a collaborative optimization unit 700, and a compound prescription determination unit 800, wherein:

[0283] The data acquisition unit 100 is used to acquire multimodal data of traditional Chinese medicine, including text data, image data, audio data, and video data.

[0284] The data processing unit 200 is used to preprocess the multimodal data of traditional Chinese medicine to obtain preprocessed multimodal data of traditional Chinese medicine.

[0285] Heterogeneous graph construction unit 300 is used to construct a heterogeneous graph based on the core representative objects and the preprocessed TCM multimodal data. The core representative objects include the core medicinal materials in the TCM multimodal data.

[0286] The data fusion unit 400 is used to input the heterogeneous graph into the graph neural network, and to fuse the preprocessed TCM multimodal data through the graph neural network to obtain TCM multimodal fused data.

[0287] The graph construction unit 500 is used to construct a knowledge graph based on multimodal fusion data of traditional Chinese medicine, and obtain a knowledge graph of traditional Chinese medicine.

[0288] The graph segmentation unit 600 is used to segment the TCM knowledge graph into subgraphs processed by multiple agents according to efficacy and medication safety, to obtain a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent. The multiple agents include two types: efficacy agents and safety agents. The efficacy agents include strategies for combining medicinal materials, and the safety agents include strategies for reviewing the safety of medicinal materials.

[0289] The collaborative optimization unit 700 is used to select medicinal materials through collaborative iterative optimization based on the medicinal material subgraph and the safety subgraph, by combining medicinal material strategies and reviewing medicinal material safety strategies, so as to obtain the optimal compound prescription recommended by traditional Chinese medicine.

[0290] The compound prescription determination unit 800 is used to input the optimal compound prescription into the fine-tuned large language model to determine the target compound prescription recommended by traditional Chinese medicine.

[0291] It should be noted that since the TCM recommendation system based on multimodal fusion in this embodiment is based on the same inventive concept as the TCM recommendation method based on multimodal fusion described above, the corresponding content in the method embodiment is also applicable to this system embodiment, and will not be described in detail here.

[0292] This application also provides an electronic device, including: at least one control processor and a memory for communicatively connecting to at least one control processor.

[0293] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0294] The non-transient software program and instructions required to implement the TCM recommendation method based on multimodal fusion in the above embodiments are stored in memory. When executed by the processor, the TCM recommendation method based on multimodal fusion in the above embodiments is executed, for example, the method described above is executed. Figure 1 The method steps S100 to S800.

[0295] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0296] This application also provides a computer-readable storage medium storing computer-executable instructions. These instructions are executed by one or more control processors, causing them to perform a multimodal fusion-based traditional Chinese medicine recommendation method described in the above-described method embodiments. For example, they can execute the methods described above. Figure 1 The functions of steps S100 to S800 in the method.

[0297] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0298] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. A method for recommending traditional Chinese medicine based on multimodal fusion, characterized in that, The method includes: Acquire multimodal data of traditional Chinese medicine that includes text, image, audio, and video data; The TCM multimodal data is preprocessed to obtain preprocessed TCM multimodal data; Based on the core representative objects and the preprocessed TCM multimodal data, a heterogeneous graph is constructed, wherein the core representative objects include the core medicinal materials in the TCM multimodal data; The heterogeneous graph is input into a graph neural network, and the preprocessed TCM multimodal data is fused through the graph neural network to obtain TCM multimodal fused data. A knowledge graph of traditional Chinese medicine is constructed based on the aforementioned multimodal fusion data of traditional Chinese medicine. The TCM knowledge graph is divided into subgraphs based on efficacy and medication safety, resulting in a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent. The multi-agent system includes two types: efficacy agents and safety agents. The efficacy agent includes strategies for combining medicinal materials, and the safety agent includes strategies for verifying the safety of medicinal materials. Specifically, this includes: The TCM knowledge graph is constructed as a main graph structure containing nodes, binary edges, hyperedges, and node feature vectors. The node feature vectors in the main graph structure are learned by multiple agents, and the node feature vectors in the learning process are enhanced by feature projection matrix to obtain the enhanced node feature vectors corresponding to each agent. Based on the enhanced node feature vector corresponding to each agent, the similarity between nodes is calculated, and a sparse adjacency matrix corresponding to each agent is constructed based on the similarity. Construct a degree matrix based on the sparse adjacency matrix corresponding to each agent; Subtracting the degree matrix from the sparse adjacency matrix yields the nonnormalized Laplace matrix. Clustering is performed on the eigenvectors in the non-normalized Laplacian matrix to obtain multiple clustering results; Each clustering result is processed as a subgraph by an agent to obtain a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent. Based on the medicinal material subgraph and the safety subgraph, the selection of medicinal materials is optimized through collaborative iterative optimization using the combined medicinal material strategy and the approved medicinal material safety strategy, resulting in the optimal compound prescription recommended by traditional Chinese medicine, specifically including: Using the aforementioned combined medicinal material strategy, medicinal materials are selected from the medicinal material sub-graph and combined to obtain the first compound prescription; Based on the safety subgraph, the medicinal materials in the first compound are subjected to safety audit using the aforementioned medicinal material safety audit strategy to obtain the second compound. The combined medicinal material strategy and the approved medicinal material safety strategy are synergistically iteratively optimized until the second compound is the same in two consecutive synergistic iterative optimizations, thus obtaining the optimal compound recommended by traditional Chinese medicine. The optimal compound is input into the finely tuned large language model to determine the target compound recommended by traditional Chinese medicine.

2. The method for recommending traditional Chinese medicine based on multimodal fusion according to claim 1, characterized in that, The preprocessing of the TCM multimodal data to obtain preprocessed TCM multimodal data includes: The text data in the TCM multimodal data is preprocessed using word embedding to obtain the first feature vector; A convolutional neural network is used to preprocess the image data in the TCM multimodal data to obtain a second feature vector; The audio data in the TCM multimodal data is preprocessed using a deep learning model to obtain a third feature vector; A convolutional neural network is used to preprocess the video data in the TCM multimodal data to obtain a fourth feature vector; The first feature vector, the second feature vector, the third feature vector, and the fourth feature vector are normalized and dimensionality reduced to obtain preprocessed TCM multimodal data.

3. The method for recommending traditional Chinese medicine based on multimodal fusion according to claim 1, characterized in that, The construction of a knowledge graph based on the multimodal fusion data of traditional Chinese medicine (TCM) to obtain a TCM knowledge graph includes: The core objects in the TCM multimodal data are selected as entity nodes, wherein the core objects include medicinal materials and symptoms in the TCM multimodal data; The aforementioned multimodal fusion data of traditional Chinese medicine is used as the attribute information of entity nodes; Relationship edges are constructed based on the relationships between the entity nodes, and initial weights are preset for the relationship edges; Based on the entity nodes, the attribute information, and the relationship edges, an initial knowledge graph is constructed; Markov decision is used to adjust the initial weights of the relation edges in the initial knowledge graph to obtain optimized weights; The optimized weights are used to replace the initial weights of the relation edges in the initial knowledge graph to construct a traditional Chinese medicine knowledge graph.

4. The method for recommending traditional Chinese medicine based on multimodal fusion according to claim 1, characterized in that, The herbal medicine sub-graph includes the relationship between symptoms and the therapeutic effects of the herbs. The method of employing the combined herbal medicine strategy involves selecting herbs from the herbal medicine sub-graph for combination to obtain a first compound prescription, including: Based on the patient's symptoms, calculate the comprehensive efficacy score of each herb in the herb sub-graph for the symptoms; Iterative adjustments are made based on the combined medicinal materials strategy, including: selecting a preset number of medicinal materials from high to low according to the comprehensive efficacy score to obtain an initial compound prescription; The medicinal materials in the initial compound prescription whose comprehensive efficacy score is greater than or equal to the first preset value shall be retained; After removing medicinal materials with a comprehensive efficacy score lower than the second preset value from the initial compound prescription, the medicinal materials are replenished to the preset number according to the comprehensive efficacy score from high to low. The first compound is obtained when the changes in the medicinal components in the compound are less than a third preset value after two consecutive rounds of iterative adjustments of the combined medicinal material strategy, or when the fluctuation of the total efficacy score of all medicinal materials in the compound is less than a fourth preset value.

5. The method for recommending traditional Chinese medicine based on multimodal fusion according to claim 1, characterized in that, The safety subgraph includes medicinal material entity nodes and incompatibility relationship edges. Based on the safety subgraph, the medicinal material safety audit strategy is used to audit the medicinal materials in the first compound to obtain the second compound, which includes: The safety strategy for reviewing medicinal materials is iteratively adjusted, including: obtaining the single-drug toxicity of each medicinal material and removing medicinal materials in the first compound whose single-drug toxicity is greater than or equal to the first safety threshold. Calculate the total toxicity of all medicinal materials in the first compound, and calculate the contribution of each medicinal material to the therapeutic effect. Based on the contribution of the therapeutic effect and the single-drug toxicity, calculate the toxicity efficiency ratio of each medicinal material. If the total toxicity is greater than the second safety threshold, then the medicinal material with the low toxicity-efficiency ratio is removed; If there is a compatibility incompatibility relationship edge in the first compound, then the medicinal material pair is obtained based on the two medicinal material entity nodes corresponding to the compatibility incompatibility relationship edge; Calculate the topological fitness of each herb in the herb pair, and remove the herb with low topological fitness in the herb pair; The second compound formula is obtained when the medicinal materials in the compound formula remain unchanged after two consecutive rounds of iterative adjustments to the safety strategy for the medicinal materials being reviewed.

6. A traditional Chinese medicine recommendation system based on multimodal fusion, characterized in that, The system includes: The data acquisition unit is used to acquire multimodal data of traditional Chinese medicine, including text data, image data, audio data, and video data. The data processing unit is used to preprocess the TCM multimodal data to obtain preprocessed TCM multimodal data. The heterogeneous graph construction unit is used to construct a heterogeneous graph based on the core representative objects and the preprocessed TCM multimodal data, wherein the core representative objects include the core medicinal materials in the TCM multimodal data; The data fusion unit is used to input the heterogeneous graph into the graph neural network, and to fuse the preprocessed TCM multimodal data through the graph neural network to obtain TCM multimodal fused data. The graph construction unit is used to construct a knowledge graph based on the TCM multimodal fusion data to obtain a TCM knowledge graph; The graph segmentation unit is used to segment the traditional Chinese medicine knowledge graph into subgraphs for multi-agent processing according to efficacy and medication safety, obtaining a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent. The multi-agent includes two types: efficacy agents and safety agents. The efficacy agent includes a medicinal material combination strategy, and the safety agent includes a medicinal material safety review strategy. Specifically, it includes: The TCM knowledge graph is constructed as a main graph structure containing nodes, binary edges, hyperedges, and node feature vectors. The node feature vectors in the main graph structure are learned by multiple agents, and the node feature vectors in the learning process are enhanced by feature projection matrix to obtain the enhanced node feature vectors corresponding to each agent. Based on the enhanced node feature vector corresponding to each agent, the similarity between nodes is calculated, and a sparse adjacency matrix corresponding to each agent is constructed based on the similarity. Construct a degree matrix based on the sparse adjacency matrix corresponding to each agent; Subtracting the degree matrix from the sparse adjacency matrix yields the nonnormalized Laplace matrix. Clustering is performed on the eigenvectors in the non-normalized Laplacian matrix to obtain multiple clustering results; Each clustering result is processed as a subgraph by an agent to obtain a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent. The collaborative optimization unit is used to select medicinal materials based on the medicinal material subgraph and the safety subgraph, through the combined medicinal material strategy and the approved medicinal material safety strategy, to obtain the optimal compound prescription recommended by traditional Chinese medicine. Specifically, it includes: Using the aforementioned combined medicinal material strategy, medicinal materials are selected from the medicinal material sub-graph and combined to obtain the first compound prescription; Based on the safety subgraph, the medicinal materials in the first compound are subjected to safety audit using the aforementioned medicinal material safety audit strategy to obtain the second compound. The combined medicinal material strategy and the approved medicinal material safety strategy are synergistically iteratively optimized until the second compound is the same in two consecutive synergistic iterative optimizations, thus obtaining the optimal compound recommended by traditional Chinese medicine. The compound determination unit is used to input the optimal compound into the fine-tuned large language model to determine the target compound recommended by traditional Chinese medicine.

7. An electronic device, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the TCM recommendation method based on multimodal fusion as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the TCM recommendation method based on multimodal fusion as described in any one of claims 1 to 5.

Images