Traditional Chinese medicine recommendation method, system and equipment based on multi-modal fusion and medium

Abstract

The invention discloses a multi-modal fusion-based traditional Chinese medicine recommendation method, system and device and a medium, and the method comprises the steps: constructing a heterogeneous graph according to a core representative object and preprocessed traditional Chinese medicine multi-modal data; inputting the heterogeneous graph into a graph neural network for data fusion to obtain traditional Chinese medicine multi-modal fusion data; constructing a knowledge graph based on the traditional Chinese medicine multi-modal fusion data to obtain a traditional Chinese medicine knowledge graph; segmenting the traditional Chinese medicine knowledge graph into sub-graphs processed by multiple agents according to the drug effect and the drug use safety, and obtaining a medicinal material sub-graph associated with a drug effect agent and a safety sub-graph associated with a safety agent; based on the medicinal material sub-graph and the safety sub-graph, collaborative iteration optimization is carried out through a medicinal material combination strategy and a medicinal material safety checking strategy to select medicinal materials, and an optimal compound recommended by the traditional Chinese medicine is obtained; and inputting the optimal compound into the fine-tuned large language model, and determining a target compound recommended by the traditional Chinese medicine. The accuracy of traditional Chinese medicine recommendation can be improved.

Description

Technical Field

[0001] This application relates to the field of intelligent medical technology, and in particular, to a traditional Chinese medicine recommendation method, system, device, and medium based on multimodal fusion. Background Art

[0002] Traditional Chinese medicine science takes the holistic concept and syndrome differentiation and treatment as the core, emphasizes individualized treatment, and its knowledge system includes multi-dimensional complex relationships such as traditional Chinese medicine, prescriptions, diseases, syndromes, and treatment methods. Traditional traditional Chinese medicine recommendations mainly rely on the experience of physicians. For physicians lacking experience, the accuracy of traditional Chinese medicine recommendations is relatively low. To solve this problem, some traditional Chinese medicine recommendation methods based on machine learning or deep learning have been proposed in the prior art.

[0003] Existing traditional Chinese medicine recommendation methods have many deficiencies. For example, (1) The traditional Chinese medicine recommendation method based on syndrome information using a graph neural network has a too high computational complexity and is difficult to apply to scenarios such as real-time recommendation and clinical diagnosis. The interpretability of the model is insufficient. The complex structures of the graph neural network and the multi-layer perception mechanism make the recommendation results difficult to be intuitively understood and explained, which may affect the trust of doctors and patients in the recommendation results. In addition, the dynamic adaptability of the model may be insufficient, lacking the ability to adapt to dynamically changing traditional Chinese medicine diagnosis and treatment data, and unable to be updated and optimized in time to cope with new diseases or treatment methods. (2) The traditional Chinese medicine recommendation method based on a multi-graph convolutional neural network has too high data dependence. This method depends on data sets of classic medical books such as Treatise on Febrile Diseases, and may not be able to fully cover the complex situations and diverse symptom combinations in modern traditional Chinese medicine diagnosis and treatment. And the model complexity is too high, resulting in a need for a large amount of computing resources. At the same time, it lacks the ability to adapt to dynamically changing traditional Chinese medicine diagnosis and treatment data and cannot be updated and optimized in time to cope with new diseases or treatment methods.

[0004] In summary, the existing traditional Chinese medicine recommendation methods have a relatively low accuracy for traditional Chinese medicine recommendations, and the existing traditional Chinese medicine recommendation methods have a relatively poor adaptability to dynamically changing data. Summary of the Invention

[0005] This application aims to propose a traditional Chinese medicine recommendation method, system, device, and medium based on multimodal fusion, which can improve the accuracy of traditional Chinese medicine recommendations and the adaptability to dynamically changing data.

[0006] In a first aspect, an embodiment of this application provides a traditional Chinese medicine recommendation method based on multimodal fusion, and the method includes: Obtain traditional Chinese medicine multimodal data including text data, image data, audio data, and video data; Preprocess the traditional Chinese medicine multimodal data to obtain preprocessed traditional Chinese medicine multimodal data; Construct a heterogeneous graph based on the core representative object and the preprocessed multi-modal traditional Chinese medicine data, where the core representative object includes the core medicinal materials in the multi-modal traditional Chinese medicine data; Input the heterogeneous graph into a graph neural network, and through the graph neural network, fuse the preprocessed multi-modal traditional Chinese medicine data to obtain multi-modal traditional Chinese medicine fusion data; Construct a knowledge graph based on the multi-modal traditional Chinese medicine fusion data to obtain a traditional Chinese medicine knowledge graph; Segment the traditional Chinese medicine knowledge graph into subgraphs processed by multiple agents according to efficacy and medication safety to obtain a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent. Among them, the multiple agents include two types: the efficacy agent and the safety agent. The efficacy agent includes a combined medicinal material strategy, and the safety agent includes a strategy for reviewing the safety of medicinal materials; Based on the medicinal material subgraph and the safety subgraph, select medicinal materials through collaborative iterative optimization using the combined medicinal material strategy and the strategy for reviewing the safety of medicinal materials to obtain the optimal compound prescription recommended by traditional Chinese medicine; Input the optimal compound prescription into the fine-tuned large language model to determine the target compound prescription recommended by traditional Chinese medicine.

[0007] Compared with the prior art, the first aspect of this application has the following beneficial effects: This method obtains traditional Chinese medicine (TCM) multi-modal data including text data, image data, audio data, and video data, preprocesses the TCM multi-modal data to obtain preprocessed TCM multi-modal data, constructs a heterogeneous graph based on the core representative objects and the preprocessed TCM multi-modal data, where the core representative objects include the core medicinal materials in the TCM multi-modal data, inputs the heterogeneous graph into a graph neural network, and through the graph neural network, fuses the preprocessed TCM multi-modal data to obtain TCM multi-modal fusion data, which can enrich the information dimension of TCM data, has high flexibility and scalability, and can meet the data application requirements in various scenarios. Then, a knowledge graph is constructed based on the TCM multi-modal fusion data to obtain a TCM knowledge graph, realizing the visual representation of TCM data. The TCM knowledge graph is then segmented into subgraphs for multi-agent processing according to the efficacy and medication safety, obtaining a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent, where the multi-agent includes two types: the efficacy agent and the safety agent. The efficacy agent includes a combined medicinal material strategy, and the safety agent includes a strategy for reviewing the safety of medicinal materials. Based on the medicinal material subgraph and the safety subgraph, the combined medicinal material strategy and the strategy for reviewing the safety of medicinal materials are used to perform collaborative iterative optimization to select medicinal materials, obtaining the optimal compound prescription recommended by TCM. By decomposing the TCM knowledge graph into subgraphs corresponding to agents and establishing a dynamic cooperation mechanism between agent interactions, the precise recommendation and safety verification of TCM compound prescriptions are realized. Finally, the optimal compound prescription is input into a fine-tuned large language model to determine the target compound prescription recommended by TCM, generating a complete compound prescription with accurate dosage, decocting method, and treatment course, improving the accuracy of TCM recommendation and the adaptability to dynamically changing data.

[0008] In some embodiments, the preprocessing of the TCM multi-modal data to obtain preprocessed TCM multi-modal data includes: Performing preprocessing on the text data in the TCM multi-modal data using word embedding to obtain a first feature vector; Performing preprocessing on the image data in the TCM multi-modal data using a convolutional neural network to obtain a second feature vector; Performing preprocessing on the audio data in the TCM multi-modal data using a deep learning model to obtain a third feature vector; Performing preprocessing on the video data in the TCM multi-modal data using a convolutional neural network to obtain a fourth feature vector; Performing normalization and dimensionality reduction on the first feature vector, the second feature vector, the third feature vector, and the fourth feature vector to obtain preprocessed TCM multi-modal data.

[0009] In some embodiments, constructing a knowledge graph based on the multi-modal fusion data of traditional Chinese medicine to obtain a traditional Chinese medicine knowledge graph includes: Selecting core objects in the multi-modal data of traditional Chinese medicine as entity nodes, where the core objects include medicinal materials and symptoms in the multi-modal data of traditional Chinese medicine; Taking the multi-modal fusion data of traditional Chinese medicine as attribute information of entity nodes; Constructing relationship edges according to the relationships between the entity nodes and presetting initial weights for the relationship edges; Constructing an initial knowledge graph according to the entity nodes, the attribute information, and the relationship edges; Adjusting the initial weights of the relationship edges in the initial knowledge graph by using Markov decision-making to obtain optimized weights; Replacing the initial weights of the relationship edges in the initial knowledge graph with the optimized weights to construct a traditional Chinese medicine knowledge graph.

[0010] In some embodiments, splitting the traditional Chinese medicine knowledge graph into subgraphs processed by multiple agents according to drug efficacy and medication safety to obtain a medicinal material subgraph associated with a drug efficacy agent and a safety subgraph associated with a safety agent includes: Constructing the traditional Chinese medicine knowledge graph into a main graph structure including nodes, binary edges, hyperedges, and node feature vectors; Learning the node feature vectors in the main graph structure through multiple agents and enhancing the node feature vectors in the learning process by using a feature projection matrix to obtain enhanced node feature vectors corresponding to each agent; Calculating the similarity between nodes according to the enhanced node feature vectors corresponding to each agent and constructing a sparse adjacency matrix corresponding to each agent based on the similarity; Constructing a degree matrix according to the sparse adjacency matrix corresponding to each agent; Subtracting the degree matrix from the sparse adjacency matrix to obtain an unnormalized Laplacian matrix; Clustering the eigenvectors in the unnormalized Laplacian matrix to obtain multiple groups of clustering results; Taking each group of clustering results as a subgraph processed by an agent to obtain a medicinal material subgraph associated with a drug efficacy agent and a safety subgraph associated with a safety agent.

[0011] In some embodiments, based on the medicinal material subgraph and the safety subgraph, synergistically and iteratively optimizing the selection of medicinal materials through the combined medicinal material strategy and the medicinal material safety review strategy to obtain an optimal traditional Chinese medicine compound prescription recommended includes: Adopt the described combined medicinal material strategy, select medicinal materials from the medicinal material sub-graph for combination to obtain the first compound prescription; Based on the safety sub-graph, adopt the strategy of auditing the safety of medicinal materials to conduct safety audits on the medicinal materials in the first compound prescription to obtain the second compound prescription; Carry out collaborative iterative optimization on the combined medicinal material strategy and the strategy of auditing the safety of medicinal materials until the second compound prescriptions in two consecutive collaborative iterative optimizations are the same, and obtain the optimal compound prescription recommended by traditional Chinese medicine.

[0012] In some embodiments, the medicinal material sub-graph includes the relationship between symptoms and the treatment effects of medicinal materials. The step of adopting the combined medicinal material strategy to select medicinal materials from the medicinal material sub-graph for combination to obtain the first compound prescription includes: According to the patient's symptoms, calculate the comprehensive curative effect scores of each medicinal material on the symptoms in the medicinal material sub-graph; Carry out iterative adjustment according to the combined medicinal material strategy, including: select a preset number of medicinal materials in descending order of the comprehensive curative effect scores to obtain an initial compound prescription; Retain the medicinal materials in the initial compound prescription whose comprehensive curative effect scores are greater than or equal to the first preset value; After removing the medicinal materials in the initial compound prescription whose comprehensive curative effect scores are less than the second preset value, supplement the medicinal materials to the preset number in descending order of the comprehensive curative effect scores; Until the change in the medicinal material components in the compound prescriptions obtained from two consecutive rounds of iterative adjustment of the combined medicinal material strategy is less than the third preset value, or the fluctuation of the total curative effect scores of all medicinal materials in the compound prescription is less than the fourth preset value, the first compound prescription is obtained.

[0013] In some embodiments, the safety sub-graph includes medicinal material entity nodes and compatibility taboo relationship edges. The step of based on the safety sub-graph, adopting the strategy of auditing the safety of medicinal materials to conduct safety audits on the medicinal materials in the first compound prescription to obtain the second compound prescription includes: Carry out iterative adjustment according to the strategy of auditing the safety of medicinal materials, including: obtain the single-drug toxicity of each medicinal material, and remove the medicinal materials in the first compound prescription whose single-drug toxicity is greater than or equal to the first safety threshold; Calculate the total toxicity of all medicinal materials in the first compound prescription, and calculate the contribution degree of the treatment effect of the medicinal materials, and calculate the toxicity efficiency ratio of each medicinal material according to the contribution degree of the treatment effect and the single-drug toxicity; If the total toxicity is greater than the second safety threshold, remove the medicinal materials with low toxicity efficiency ratio; If there are compatibility taboo relationship edges in the first compound prescription, obtain the medicinal material pairs according to the two medicinal material entity nodes corresponding to the compatibility taboo relationship edges; Calculate the topological fitness of each medicinal material in the medicinal material pair, and remove the medicinal materials with low topological fitness in the medicinal material pair; Until the herbs in the compound remain unchanged after two consecutive iterative adjustments of the strategy for reviewing the safety of herbs, a second compound is obtained.

[0014] Second, the embodiments of the present application also provide a traditional Chinese medicine recommendation system based on multimodal fusion. The system includes: A data acquisition unit for acquiring traditional Chinese medicine multimodal data including text data, image data, audio data, and video data; A data processing unit for preprocessing the traditional Chinese medicine multimodal data to obtain preprocessed traditional Chinese medicine multimodal data; A heterogeneous graph construction unit for constructing a heterogeneous graph according to the core representative objects and the preprocessed traditional Chinese medicine multimodal data, where the core representative objects include the core herbs in the traditional Chinese medicine multimodal data; A data fusion unit for inputting the heterogeneous graph into a graph neural network, and fusing the preprocessed traditional Chinese medicine multimodal data through the graph neural network to obtain traditional Chinese medicine multimodal fusion data; A knowledge graph construction unit for constructing a knowledge graph based on the traditional Chinese medicine multimodal fusion data to obtain a traditional Chinese medicine knowledge graph; A knowledge graph segmentation unit for segmenting the traditional Chinese medicine knowledge graph into subgraphs processed by multiple agents according to the efficacy and medication safety, to obtain a herb subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent, where the multiple agents include two types: an efficacy agent and a safety agent, the efficacy agent includes a combined herb strategy, and the safety agent includes a strategy for reviewing the safety of herbs; A collaborative optimization unit for collaboratively and iteratively optimizing the selection of herbs based on the herb subgraph and the safety subgraph through the combined herb strategy and the strategy for reviewing the safety of herbs to obtain the optimal compound for traditional Chinese medicine recommendation; A compound determination unit for inputting the optimal compound into a fine-tuned large language model to determine the target compound for traditional Chinese medicine recommendation.

[0015] Third, the embodiments of the present application also provide an electronic device, including at least one control processor and a memory communicatively connected to the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a traditional Chinese medicine recommendation method based on multimodal fusion as described above.

[0016] Fourthly, an embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions for causing a computer to execute a traditional Chinese medicine recommendation method based on multi-modal fusion as described above.

[0017] It can be understood that the beneficial effects of the above second to fourth aspects compared with the related art are the same as those of the above first aspect compared with the related art. For the relevant descriptions, reference can be made to the above first aspect and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the accompanying drawings, where: Figure 1 is a schematic flowchart of an embodiment of a traditional Chinese medicine recommendation method based on multi-modal fusion provided by the present application; Figure 2 is a schematic overall method flowchart of the best embodiment of a traditional Chinese medicine recommendation method based on multi-modal fusion provided by the present application; Figure 3 is a schematic diagram of multi-modal data preprocessing in the best embodiment of a traditional Chinese medicine recommendation method based on multi-modal fusion provided by the present application; Figure 4 is a schematic flowchart of a traditional Chinese medicine multi-modal data fusion module in the best embodiment of a traditional Chinese medicine recommendation method based on multi-modal fusion provided by the present application; Figure 5 is a schematic flowchart of knowledge graph construction based on reinforcement learning in the best embodiment of a traditional Chinese medicine recommendation method based on multi-modal fusion provided by the present application; Figure 6 is a schematic diagram of a two-layer divide-and-conquer architecture in the best embodiment of a traditional Chinese medicine recommendation method based on multi-modal fusion provided by the present application; Figure 7 is a schematic structural diagram of an embodiment of a traditional Chinese medicine recommendation system based on multi-modal fusion provided by the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and should not be construed as a limitation of the present application.

[0020] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0021] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0022] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0023] First, some nouns involved in this application are analyzed: Text cleaning technology: refers to the preprocessing of raw text data, removing useless information, correcting errors, unifying formats, etc., in order to improve the quality of the text and make it more suitable for subsequent analysis and processing.

[0024] Principal component analysis (PCA): A commonly used dimensionality reduction technique. Its principle is to perform eigendecomposition of the covariance matrix of the data, find several orthogonal directions with the largest variance in the data (i.e., principal components), and then project the original data onto these principal component directions, thereby reducing the data dimension while retaining the main information of the data as much as possible.

[0025] Adjacency matrix: The adjacency matrix is ​​a way to represent a graph, using a two-dimensional matrix to represent the connection relationship between nodes in the graph.

[0026] Heterogeneous graph: A heterogeneous graph means that the nodes and edges in the graph can be of different types, and there can be specific semantic relationships between different types.

[0027] Degree matrix: The degree matrix is ​​a diagonal matrix associated with a graph, where each diagonal element represents the degree of the corresponding node (i.e., the number of edges connected to the node). In the calculation process of graph neural networks, the degree matrix is ​​usually used to normalize the feature aggregation of nodes to avoid excessive amplification or reduction of the feature information of some nodes during the propagation process due to large differences in node degrees, which helps to improve the stability and performance of the model.

[0028] Traditional Chinese Medicine Knowledge Graph: It is a structured knowledge system that integrates data such as traditional Chinese medicine theory, medicinal materials, and prescriptions. By establishing associations between concepts (such as the relationship between medicinal materials and curative effects), it reveals internal laws. Its core is to express the complex relationships in the field of traditional Chinese medicine in the form of a semantic network, which is used in scenarios such as auxiliary diagnosis and drug compatibility analysis, and supports systematic exploration of research and applications.

[0029] Agent: It refers to an entity that can perceive the environment and take autonomous actions to achieve goals, usually possessing learning, reasoning, and decision-making abilities.

[0030] Correlation Matrix: It is a two-dimensional matrix that quantifies the similarity or difference between multiple groups of objects pairwise. Each element represents the degree of association between the corresponding sets. By systematically comparing objects such as shapes and point sets, it can reveal the overall association law, and is commonly used in pattern recognition or data analysis to assist in understanding complex relationships from a global perspective.

[0031] Weight Matrix: It is a structured tool used to quantify the importance or association strength between data attributes, usually represented in the form of a two-dimensional matrix. Each of its elements reflects the weight value between different feature pairs. By adjusting these weights, the attention of the model to key features can be optimized, and it is commonly used in feature selection, data analysis, or model training to assist in mining the core association patterns between data.

[0032] Monarch, Minister, Assistant, and Guide: It is the core principle of traditional Chinese medicine compatibility. By clarifying the hierarchical role division of drugs in a prescription (the main drug "monarch", the assistant drug "minister", the synergistic drug "assistant", and the harmonizing drug "guide"), a multi-dimensional collaborative system is constructed. Its core lies in integrating different medicinal effects with a clear primary and secondary structure, not only strengthening the core therapeutic effect but also balancing the conflict of medicinal properties, while taking into account secondary symptoms and the regulation of drug toxicity, thereby optimizing the accuracy and safety of treating complex diseases and reflecting the unique thinking of "overall regulation" in traditional Chinese medicine.

[0033] Pagerank: It is a static weight distribution algorithm based on a graph structure that evaluates the global importance by analyzing the link relationships between nodes. Its core idea is that "the node linked by more high-weight nodes is more important", and it is commonly used in scenarios such as web page ranking and social network influence analysis.

[0034] Dynamic Adaptability: It refers to whether a method or system can adjust the recommendation strategy in real time according to the changes in the patient's condition, individual differences, and new data input.

[0035] Compound Prescription: It refers to a combination of traditional Chinese medicines, such as Danggui Buxue Decoction and Liuwei Dihuang Pills.

[0036] Syndrome Type: It refers to the classification of traditional Chinese medicine syndrome differentiation (disease phenotype), such as qi and blood deficiency syndrome and kidney yin deficiency syndrome.

[0037] Pathway: It refers to a biological pathway or signal network, such as the tumor necrosis factor (TNF) signaling pathway.

[0038] Existing traditional Chinese medicine (TCM) recommendation methods have many deficiencies. For example, (1) the TCM recommendation method based on syndrome information using a graph neural network has a too high computational complexity and is difficult to apply to scenarios such as real-time recommendation and clinical diagnosis. The interpretability of the model is insufficient. The complex structures of the graph neural network and the multi-layer perceptron make it difficult to intuitively understand and interpret the recommendation results, which may affect the trust of doctors and patients in the recommendation results. In addition, the dynamic adaptability of the model may be insufficient, lacking the ability to adapt to dynamically changing TCM diagnosis and treatment data and unable to update and optimize in a timely manner to cope with new diseases or treatment methods. (2) The TCM recommendation method based on a multi-graph convolutional neural network has too high data dependence. This method relies on datasets from classic medical books such as Treatise on Febrile and Miscellaneous Diseases and may not fully cover the complex situations and diverse symptom combinations in modern TCM diagnosis and treatment. Moreover, the model complexity is too high, requiring a large amount of computing resources. At the same time, it lacks the ability to adapt to dynamically changing TCM diagnosis and treatment data and cannot update and optimize in a timely manner to cope with new diseases or treatment methods.

[0039] To solve the problems that the existing TCM recommendation methods have relatively low accuracy in TCM recommendation and relatively poor adaptability to dynamically changing data.

[0040] Referring to Figure 1 , the flowchart of the TCM recommendation method based on multi-modal fusion provided by the embodiments of this application. This TCM recommendation method based on multi-modal fusion is applied to an electronic device, which can be a server or a mobile terminal, etc. As Figure 1 shown, this TCM recommendation method based on multi-modal fusion may include the following steps: Step S100: Obtain TCM multi-modal data including text data, image data, audio data, and video data; Step S200: Preprocess the TCM multi-modal data to obtain preprocessed TCM multi-modal data; Step S300: Construct a heterogeneous graph according to the core representative objects and the preprocessed TCM multi-modal data, where the core representative objects include the core medicinal materials in the TCM multi-modal data; Step S400: Input the heterogeneous graph into a graph neural network, and fuse the preprocessed TCM multi-modal data through the graph neural network to obtain TCM multi-modal fusion data; Step S500: Construct a knowledge graph based on the TCM multi-modal fusion data to obtain a TCM knowledge graph; Step S600: Segment the traditional Chinese medicine knowledge graph into subgraphs for multi-agent processing according to the medicinal effects and medication safety, obtaining a medicinal material subgraph associated with the medicinal effect agent and a safety subgraph associated with the safety agent. Among them, the multi-agent includes two types: the medicinal effect agent and the safety agent. The medicinal effect agent includes a combined medicinal material strategy, and the safety agent includes a strategy for reviewing the safety of medicinal materials. Step S700: Based on the medicinal material subgraph and the safety subgraph, select medicinal materials through collaborative iterative optimization using the combined medicinal material strategy and the strategy for reviewing the safety of medicinal materials, obtaining the optimal compound prescription recommended by traditional Chinese medicine. Step S800: Input the optimal compound prescription into the fine-tuned large language model to determine the target compound prescription recommended by traditional Chinese medicine.

[0041] In this embodiment, by obtaining traditional Chinese medicine multi-modal data including text data, image data, audio data, and video data, preprocessing the traditional Chinese medicine multi-modal data to obtain preprocessed traditional Chinese medicine multi-modal data, and constructing a heterogeneous graph according to the core representative objects and the preprocessed traditional Chinese medicine multi-modal data. Among them, the core representative objects include the core medicinal materials in the traditional Chinese medicine multi-modal data. Input the heterogeneous graph into a graph neural network, and through the graph neural network, fuse the preprocessed traditional Chinese medicine multi-modal data to obtain traditional Chinese medicine multi-modal fusion data, which can enrich the information dimension of traditional Chinese medicine data, has high flexibility and scalability, and can meet the data application requirements in various scenarios. Then, based on the traditional Chinese medicine multi-modal fusion data, construct a knowledge graph to obtain the traditional Chinese medicine knowledge graph, realizing the visual representation of traditional Chinese medicine data. Then segment the traditional Chinese medicine knowledge graph into subgraphs for multi-agent processing according to the medicinal effects and medication safety, obtaining a medicinal material subgraph associated with the medicinal effect agent and a safety subgraph associated with the safety agent. Among them, the multi-agent includes two types: the medicinal effect agent and the safety agent. The medicinal effect agent includes a combined medicinal material strategy, and the safety agent includes a strategy for reviewing the safety of medicinal materials. Based on the medicinal material subgraph and the safety subgraph, select medicinal materials through collaborative iterative optimization using the combined medicinal material strategy and the strategy for reviewing the safety of medicinal materials, obtaining the optimal compound prescription recommended by traditional Chinese medicine. By decomposing the traditional Chinese medicine knowledge graph into subgraphs corresponding to agents and establishing a dynamic cooperation mechanism between agent interactions, accurate recommendation and safety verification of traditional Chinese medicine compound prescriptions are realized. Finally, input the optimal compound prescription into the fine-tuned large language model to determine the target compound prescription recommended by traditional Chinese medicine, generating a complete compound prescription with accurate dosage, decocting method, and treatment course, improving the accuracy of traditional Chinese medicine recommendation and the adaptability to dynamically changing data.

[0042] The above preprocessing of the traditional Chinese medicine multi-modal data to obtain the preprocessed traditional Chinese medicine multi-modal data can be to process each type of modal data in the traditional Chinese medicine multi-modal data in different ways to obtain the preprocessed traditional Chinese medicine multi-modal data. The processing methods can be feature extraction, normalization processing, etc.

[0043] Based on the above core representative objects and preprocessed traditional Chinese medicine multi-modal data, an heterogeneous graph can be constructed by taking the core representative objects as entity nodes, the preprocessed traditional Chinese medicine multi-modal data as modal nodes, constructing edges connecting entity nodes and modal nodes, and constructing an heterogeneous graph according to the relationships between nodes and edges.

[0044] The above traditional Chinese medicine multi-modal fusion data can be the feature vectors fused by a graph neural network as the traditional Chinese medicine multi-modal fusion data.

[0045] Inputting the optimal compound prescription into the fine-tuned large language model to determine the target compound prescription for traditional Chinese medicine recommendation can be to fine-tune the pre-trained large language model using the Chinese Pharmacopoeia, and then input the optimal compound prescription into the fine-tuned large language model, which can complete the supplementation of the optimal compound prescription to determine the target compound prescription for traditional Chinese medicine recommendation. The pre-trained large language model can be obtained by training the large language model using existing data sets.

[0046] In some embodiments, preprocessing the traditional Chinese medicine multi-modal data to obtain preprocessed traditional Chinese medicine multi-modal data includes: Preprocessing the text data in the traditional Chinese medicine multi-modal data using word embedding to obtain a first feature vector; Preprocessing the image data in the traditional Chinese medicine multi-modal data using a convolutional neural network to obtain a second feature vector; Preprocessing the audio data in the traditional Chinese medicine multi-modal data using a deep learning model to obtain a third feature vector; Preprocessing the video data in the traditional Chinese medicine multi-modal data using a convolutional neural network to obtain a fourth feature vector; Performing normalization and dimensionality reduction processing on the first feature vector, the second feature vector, the third feature vector, and the fourth feature vector to obtain preprocessed traditional Chinese medicine multi-modal data.

[0047] In this embodiment, the text data in the multi-modal data of traditional Chinese medicine is preprocessed by using word embedding to obtain the first feature vector; the image data in the multi-modal data of traditional Chinese medicine is preprocessed by using a convolutional neural network to obtain the second feature vector; the audio data in the multi-modal data of traditional Chinese medicine is preprocessed by using a deep learning model to obtain the third feature vector; the video data in the multi-modal data of traditional Chinese medicine is preprocessed by using a convolutional neural network to obtain the fourth feature vector; the first feature vector, the second feature vector, the third feature vector, and the fourth feature vector are normalized and dimension-reduced to obtain the preprocessed multi-modal data of traditional Chinese medicine. In this way, different data processing methods are used for different modal data, which can obtain more accurate feature data, reduce the data processing volume during the later multi-modal data fusion, improve the data processing efficiency, and enrich the information dimension of the multi-modal traditional Chinese medicine data in the extraction and characterization of the multi-modal traditional Chinese medicine data.

[0048] In some embodiments, a knowledge graph is constructed based on the multi-modal fusion data of traditional Chinese medicine to obtain a traditional Chinese medicine knowledge graph, including: Select the core objects in the multi-modal data of traditional Chinese medicine as entity nodes, where the core objects include the medicinal materials and symptoms in the multi-modal data of traditional Chinese medicine; Use the multi-modal fusion data of traditional Chinese medicine as the attribute information of the entity nodes; Construct relationship edges according to the relationships between the entity nodes, and preset initial weights for the relationship edges; Construct an initial knowledge graph based on the entity nodes, attribute information, and relationship edges; Use Markov decision-making to adjust the initial weights of the relationship edges in the initial knowledge graph to obtain optimized weights; Replace the initial weights of the relationship edges in the initial knowledge graph with the optimized weights to construct a traditional Chinese medicine knowledge graph.

[0049] In this embodiment, by selecting the core objects in the multi-modal data of traditional Chinese medicine as entity nodes, where the core objects include the medicinal materials and symptoms in the multi-modal data of traditional Chinese medicine; using the multi-modal fusion data of traditional Chinese medicine as the attribute information of the entity nodes; constructing relationship edges according to the relationships between the entity nodes, and presetting initial weights for the relationship edges; constructing an initial knowledge graph based on the entity nodes, attribute information, and relationship edges; using Markov decision-making to adjust the initial weights of the relationship edges in the initial knowledge graph to obtain optimized weights; replacing the initial weights of the relationship edges in the initial knowledge graph with the optimized weights to construct a traditional Chinese medicine knowledge graph. In this way, the visualization representation of traditional Chinese medicine data is realized by using the knowledge graph, and the weight allocation and path reasoning efficiency of the entity relationships in the knowledge graph are optimized by means of the dynamic decision-making ability of reinforcement learning (i.e., Markov decision-making).

[0050] In some embodiments, the traditional Chinese medicine knowledge graph is segmented into subgraphs processed by multiple agents according to medicinal effects and medication safety, obtaining a medicinal material subgraph associated with a medicinal effect agent and a safety subgraph associated with a safety agent, including: Construct the traditional Chinese medicine knowledge graph into a main graph structure including nodes, binary edges, hyperedges, and node feature vectors; Let multiple agents learn the node feature vectors in the main graph structure, and use a feature projection matrix to enhance the node feature vectors during the learning process to obtain enhanced node feature vectors corresponding to each agent; Calculate the similarity between nodes according to the enhanced node feature vectors corresponding to each agent, and construct a sparse adjacency matrix corresponding to each agent based on the similarity; Construct a degree matrix according to the sparse adjacency matrix corresponding to each agent; Subtract the degree matrix from the sparse adjacency matrix to obtain an unnormalized Laplacian matrix; Cluster the feature vectors in the unnormalized Laplacian matrix to obtain multiple groups of clustering results; Take each group of clustering results as a subgraph processed by an agent to obtain a medicinal material subgraph associated with a medicinal effect agent and a safety subgraph associated with a safety agent.

[0051] In this embodiment, by constructing the traditional Chinese medicine knowledge graph into a main graph structure including nodes, binary edges, hyperedges, and node feature vectors; letting multiple agents learn the node feature vectors in the main graph structure, and using a feature projection matrix to enhance the node feature vectors during the learning process to obtain enhanced node feature vectors corresponding to each agent; calculating the similarity between nodes according to the enhanced node feature vectors corresponding to each agent, and constructing a sparse adjacency matrix corresponding to each agent based on the similarity; constructing a degree matrix according to the sparse adjacency matrix corresponding to each agent; subtracting the degree matrix from the sparse adjacency matrix to obtain an unnormalized Laplacian matrix; clustering the feature vectors in the unnormalized Laplacian matrix to obtain multiple groups of clustering results; taking each group of clustering results as a subgraph processed by an agent to obtain a medicinal material subgraph associated with a medicinal effect agent and a safety subgraph associated with a safety agent. In this way, multi-dimensional structural cognition is achieved. The safety agent constructs a taboo subgraph with highly similar connections in the projection space by strengthening the similarity of taboo features; the medicinal effect agent enhances the similarity of meridian tropism features to form dense connections of nodes with similar meridian tropisms. The differential mechanism enables each agent to accurately capture task-related topological patterns in its exclusive feature space, laying a good data foundation for later traditional Chinese medicine recommendation.

[0052] In some embodiments, based on the medicinal material sub-graph and the safety sub-graph, the medicinal materials are selected through collaborative iterative optimization by combining the medicinal material strategy and the strategy for reviewing the safety of medicinal materials, and the optimal compound prescription recommended by traditional Chinese medicine is obtained, including: Adopt the medicinal material combination strategy to select and combine medicinal materials from the medicinal material sub-graph to obtain the first compound prescription; Based on the safety sub-graph, adopt the strategy for reviewing the safety of medicinal materials to conduct safety reviews on the medicinal materials in the first compound prescription to obtain the second compound prescription; Perform collaborative iterative optimization on the medicinal material combination strategy and the strategy for reviewing the safety of medicinal materials until the second compound prescriptions in two consecutive collaborative iterative optimizations are the same, and the optimal compound prescription recommended by traditional Chinese medicine is obtained.

[0053] In this embodiment, by adopting the medicinal material combination strategy, medicinal materials are selected and combined from the medicinal material sub-graph to obtain the first compound prescription; based on the safety sub-graph, the strategy for reviewing the safety of medicinal materials is adopted to conduct safety reviews on the medicinal materials in the first compound prescription to obtain the second compound prescription; perform collaborative iterative optimization on the medicinal material combination strategy and the strategy for reviewing the safety of medicinal materials until the second compound prescriptions in two consecutive collaborative iterative optimizations are the same, and the optimal compound prescription recommended by traditional Chinese medicine is obtained. In this way, through collaborative iterative optimization by combining the medicinal material combination strategy and the strategy for reviewing the safety of medicinal materials, that is, introducing an auxiliary retrieval scheme with a two-layer divide-and-conquer architecture, not only greatly reduces the spatio-temporal range of graph traversal retrieval, significantly reduces the problem of the demand for computing resources in traditional Chinese medicine recommendation schemes, but also improves the accuracy of traditional Chinese medicine recommendation.

[0054] In some embodiments, the medicinal material sub-graph includes the relationship between symptoms and the treatment effects of medicinal materials. Adopt the medicinal material combination strategy to select and combine medicinal materials from the medicinal material sub-graph to obtain the first compound prescription, including: According to the patient's symptoms, calculate the comprehensive efficacy scores of each medicinal material in the medicinal material sub-graph for the symptoms; Perform iterative adjustment according to the medicinal material combination strategy, including: select a preset number of medicinal materials in descending order of the comprehensive efficacy scores to obtain the initial compound prescription; Retain the medicinal materials in the initial compound prescription whose comprehensive efficacy scores are greater than or equal to the first preset value; After removing the medicinal materials in the initial compound prescription whose comprehensive efficacy scores are less than the second preset value, and supplement the medicinal materials to the preset number in descending order of the comprehensive efficacy scores; Until the change in the medicinal material components in the compound prescriptions in two consecutive rounds of iterative adjustment by the medicinal material combination strategy is less than the third preset value, or the fluctuation of the total efficacy scores of all the medicinal materials in the compound prescription is less than the fourth preset value, the first compound prescription is obtained.

[0055] In this embodiment, by calculating the comprehensive efficacy score of each medicinal material in the sub-graph of medicinal materials for the symptoms according to the patient's symptoms; iterative adjustment is performed according to the combined medicinal material strategy, including: selecting a preset number of medicinal materials from high to low according to the comprehensive efficacy score to obtain an initial compound prescription; retaining the medicinal materials in the initial compound prescription whose comprehensive efficacy score is greater than or equal to the first preset value; removing the medicinal materials in the initial compound prescription whose comprehensive efficacy score is less than the second preset value, and then supplementing to the preset number of medicinal materials from high to low according to the comprehensive efficacy score; until the change in the medicinal material components in the compound prescriptions obtained from two consecutive rounds of iterative adjustment of the combined medicinal material strategy is less than the third preset value, or the fluctuation of the total efficacy score of all medicinal materials in the compound prescription is less than the fourth preset value, the first compound prescription is obtained. In this way, the medicinal materials related to the patient's symptoms are screened through the comprehensive efficacy score, meeting the rules of monarch, minister, assistant, and envoy in traditional Chinese medicine compound prescriptions, making the obtained compound prescription have good efficacy, so as to improve the accuracy of traditional Chinese medicine recommendation.

[0056] The values of the above first preset value, second preset value, third preset value, and fourth preset value can be changed according to the actual situation, and this embodiment does not make specific limitations.

[0057] In some embodiments, the safety sub-graph includes medicinal material entity nodes and compatibility taboo relationship edges. Based on the safety sub-graph, the medicinal materials in the first compound prescription are subjected to safety review using the medicinal material safety review strategy to obtain the second compound prescription, including: Iterative adjustment is performed according to the medicinal material safety review strategy, including: obtaining the single-drug toxicity of each medicinal material, and removing the medicinal materials in the first compound prescription whose single-drug toxicity is greater than or equal to the first safety threshold; Calculating the total toxicity of all medicinal materials in the first compound prescription, and calculating the contribution degree of the medicinal materials to the treatment effect, and calculating the toxicity efficiency ratio of each medicinal material according to the contribution degree of the treatment effect and the single-drug toxicity; If the total toxicity is greater than the second safety threshold, remove the medicinal materials with low toxicity efficiency ratio; If there are compatibility taboo relationship edges in the first compound prescription, obtain the medicinal material pairs according to the two medicinal material entity nodes corresponding to the compatibility taboo relationship edges; Calculating the topological fitness of each medicinal material in the medicinal material pair, and removing the medicinal material with low topological fitness in the medicinal material pair; Until the medicinal materials in the compound prescriptions obtained from two consecutive rounds of iterative adjustment of the medicinal material safety review strategy remain unchanged, the second compound prescription is obtained.

[0058] In this embodiment, iterative adjustment is performed according to the strategy for reviewing the safety of medicinal materials, including: obtaining the single-herb toxicity of each medicinal material, and removing the medicinal materials in the first compound formula whose single-herb toxicity is greater than or equal to the first safety threshold; calculating the total toxicity of all the medicinal materials in the first compound formula, and calculating the contribution degree of the medicinal materials to the therapeutic effect, and calculating the toxicity efficiency ratio of each medicinal material according to the contribution degree of the therapeutic effect and the single-herb toxicity; if the total toxicity is greater than the second safety threshold, remove the medicinal materials with low toxicity efficiency ratio; if there is a compatibility taboo relationship edge in the first compound formula, obtain the medicinal material pair according to the two medicinal material entity nodes corresponding to the compatibility taboo relationship edge; calculate the topological fitness of each medicinal material in the medicinal material pair, and remove the medicinal materials with low topological fitness in the medicinal material pair; until the medicinal materials in the compound formula remain unchanged after two consecutive rounds of iterative adjustment of the strategy for reviewing the safety of medicinal materials, and the second compound formula is obtained. In this way, according to the taboo relationship between medicinal materials and the properties such as the toxicity of the medicinal materials themselves, the conflicts between the medicinal materials themselves and between the medicinal materials are calculated, and the conflicting medicinal materials are removed, and finally a safe compound formula is obtained to improve the accuracy of traditional Chinese medicine recommendation.

[0059] The numerical values of the above first safety threshold and second safety threshold can be changed according to the actual situation, and no specific limitation is made in this embodiment.

[0060] The above removal of the medicinal materials with low toxicity efficiency ratio can be to remove only the medicinal material with the lowest toxicity efficiency ratio, or to remove multiple medicinal materials with low toxicity efficiency ratio. The specific number of medicinal materials to be removed can be changed according to the actual situation, and no specific limitation is made in this embodiment.

[0061] The above removal of the medicinal materials with low topological fitness in the medicinal material pair can be to remove only the medicinal material with the lowest topological fitness, or to remove multiple medicinal materials with low topological fitness. The specific number of medicinal materials to be removed can be changed according to the actual situation, and no specific limitation is made in this embodiment.

[0062] For the convenience of understanding by those skilled in the art, a set of best embodiments are provided below: This embodiment proposes an auxiliary retrieval scheme based on a graph neural network for multi-modal traditional Chinese medicine data fusion, reinforcement learning-based knowledge graph representation, and a two-layer divide-and-conquer architecture. This scheme preprocesses different modalities of traditional Chinese medicine data, maps them into a unified dimension and semantic space, and based on the graph neural network, fuses the preprocessed multi-modal data to obtain highly representative core traditional Chinese medicine data information, extracting and integrating key information and data from a large amount of multi-modal traditional Chinese medicine data. According to the core objects and data information in the corresponding traditional Chinese medicine data, a complex traditional Chinese medicine knowledge graph is constructed, and the constructed traditional Chinese medicine knowledge graph can be visually adjusted dynamically. According to the strategy of reinforcement learning, the relevant weights of the edges between entities are dynamically adjusted. Finally, with the assistance of the two-layer divide-and-conquer architecture and the large language model, an accurate and efficient drug diagnosis and treatment assistance scheme is provided. Compared with traditional methods, the method in this embodiment effectively integrates a large amount of traditional Chinese medicine data and explores the potential of data through multi-modal fusion, and combines the strategy of reinforcement learning to construct a complex but detailed traditional Chinese medicine knowledge graph. Combining the two-layer divide-and-conquer architecture with the large language model, it realizes the efficient and accurate recommendation of traditional Chinese medicine. This method can also make up for the defects that traditional graph neural network methods are difficult to be intuitively understood and the dynamic adaptability of the model may be insufficient. Refer to Figure 2 , the technical solution of this embodiment specifically includes the following content: I. A multi-modal traditional Chinese medicine data fusion and representation scheme based on a graph neural network.

[0063] In order to effectively extract key information from traditional Chinese medicine data during processing. This embodiment proposes a multi-modal traditional Chinese medicine data fusion and representation scheme based on a graph neural network. This scheme realizes the extraction and representation tasks of multi-modal traditional Chinese medicine data through the strategy of multi-modal data fusion by the graph neural network. Its core lies in using the graph neural network combined with multi-modal fusion technology to convert unstructured multi-modal traditional Chinese medicine data into structured data, which not only enriches the information dimension of traditional Chinese medicine data but also deeply explores the potential value of traditional Chinese medicine data. This scheme not only extracts the core information from a large amount of multi-modal traditional Chinese medicine data but also has high flexibility and scalability, meeting the data application requirements in various scenarios.

[0064] 1. Introduction to multi-modal data preprocessing.

[0065] Multi-modal data preprocessing is the first step of the scheme in this embodiment. Its core idea is to preprocess a large amount of multi-modal data such as traditional Chinese medicine texts, images, and audio, reduce data redundancy, and retain valuable content as much as possible. At the same time, it effectively reduces the data processing volume for multi-modal data fusion through the graph neural network (GNN) in the subsequent process and improves the data processing efficiency. Refer to Figure 3 , the specific process is as follows: (1)Data input: Receive multi-modal data of traditional Chinese medicine, including text data such as classical literature, medical records, treatises, etc., traditional Chinese medicine image data, audio data, and video data.

[0066] (2)Data preprocessing: Preprocess data of different modalities separately, and map the features extracted from different modalities to the same semantic space. The specific preprocessing methods are as follows: 1) Text data: Use text cleaning technology to remove special symbols such as "@", "#" in the text content and filter some meaningless stop words such as "not as good as", "compared with", etc., to standardize the text data. Then use word segmentation technology and natural language processing technology (NLP) to extract and classify the core words in the text data. Finally, use word embedding technology to convert the processed text data into a numerical feature vector representation (i.e., the first feature vector).

[0067] 2) Image data: Normalize the size of the image data, uniformly adjust it to the same size, and standardize the color to meet the requirements of model input. Then, use a convolutional neural network (CNN) model to extract the feature vector of the image data (i.e., the second feature vector).

[0068] 3) Audio data: Unify the sampling rate of the audio data to a fixed value to ensure the consistency of the sampling rates of different audio files, and then perform noise reduction processing on the audio data to effectively remove background noise and make the audio content clear. Finally, use the pre-trained deep learning model OpenL3 to extract the audio feature vector of a fixed length (i.e., the third feature vector).

[0069] 4) Video data: Decode the video into a series of consecutive video frames, and then extract key frames according to the method of sampling at intervals. The key frame feature vector is also extracted using a convolutional neural network (CNN) model, and the semantic feature vectors of the key frames are extracted respectively as the video feature vector (i.e., the fourth feature vector).

[0070] (3)L2 normalization processing: Perform L2 normalization processing on the feature vectors extracted from each modality to eliminate the scale difference between the features of different modalities. The formula is: ; where represents the feature vector after L2 normalization processing, represents the elements in the feature vector.

[0071] (4)Dimensionality reduction processing: Perform principal component analysis (PCA) linear dimensionality reduction on each individual modality to unify the features of different modalities to the same dimension for subsequent processing.

[0072] (5) Data transfer: Transfer the processed feature vectors (i.e., the preprocessed traditional Chinese medicine multimodal data) to the traditional Chinese medicine multimodal data fusion module.

[0073] 2. Introduction to traditional Chinese medicine multimodal data fusion.

[0074] Refer to Figure 4 for the schematic diagram of the process of the traditional Chinese medicine multimodal data fusion module. The core method of traditional Chinese medicine multimodal data fusion is to use GNN to fuse the feature vectors of each modality extracted from the multimodal data preprocessing section, providing relevant information for the subsequent construction of the traditional Chinese medicine knowledge graph.

[0075] 2.1. Construction of the heterogeneous graph.

[0076] The construction of the heterogeneous graph is the first step of the traditional Chinese medicine multimodal data fusion solution, mainly responsible for organizing each traditional Chinese medicine entity and its associated multimodal data (image, text, audio data, etc.) into a graph structure, providing a basis for subsequent feature aggregation and multimodal data fusion based on the graph neural network (GNN). The specific process is as follows: (1) The nodes of the graph are defined as follows, and a unique index is assigned to each entity node: 1) Definition of entity nodes: Take the core representative objects (e.g., core medicinal materials such as Ganoderma lucidum and Pilose antler) as entity nodes.

[0077] 2) Definition of modality nodes: Take the feature vectors of each modality as the corresponding modality nodes.

[0078] (2) Definition of the edges of the graph: Connect the entity nodes and the corresponding modality nodes.

[0079] (3) Construction of the adjacency relationship: According to the relationship between the nodes and the edges, define and construct the corresponding adjacency matrix for each edge. The rows and columns of the matrix represent different types of nodes, and the value of 1 indicates that the nodes are connected, and the value of 0 indicates that the nodes are not connected, providing a basis for subsequent multimodal data fusion using the graph neural network.

[0080] 2.2. GNN realizes the fusion of multimodal data.

[0081] This is the second step of the traditional Chinese medicine multimodal data fusion solution, mainly responsible for using the graph neural network to realize the fusion of multimodal data. GNN can effectively capture the complex correlations between multimodal data by analyzing the node and edge relationships in the graph structure, thereby realizing the deep fusion of features. The specific process: (1) Definition of the message passing rules of the heterogeneous graph.

[0082] Forward transfer: The modality nodes transfer to the entity nodes for forming the multimodal joint representation.

[0083] (2)Feature initialization and aggregation.

[0084] 1) Feature initialization: The feature vectors of each node (including entity nodes and modality nodes) in the heterogeneous graph are used as the initial input. The feature vectors of entity nodes can be their basic attributes (such as name, category, etc.), while the feature vectors of modality nodes are the feature vectors extracted from multimodal data (such as images, texts, and audios, etc.).

[0085] 2) Feature aggregation: Through the aggregation mechanism of the GNN, that is, the feature vectors of adjacent nodes are weighted summed or concatenated to generate a new feature representation. This process can capture the correlation information between nodes and provide a basis for subsequent feature fusion.

[0086] (3)Construction of the graph neural network.

[0087] Graph Convolutional Layer (GCN): The graph convolutional layer is used to update the node features. The feature vector of each node interacts with the features of adjacent nodes through the adjacency matrix to capture local structural information. The formula is: ; Among them, represents the adjacency matrix with self-connections added, represents the degree matrix, represents the node feature matrix of the th layer of the graph neural network, represents the weight matrix, represents the activation function, represents the node feature matrix of the th layer of the graph neural network.

[0088] (4)Information transfer and fusion.

[0089] 1) Information transfer: Through the propagation mechanism of the GNN, the feature information of the modality nodes is transferred to the entity nodes. The adjacency matrix records the connection relationships between nodes, and the GNN uses this structure to achieve the layer-by-layer transfer of features.

[0090] 2) Feature fusion: At the entity node level, the feature vectors from different modalities are fused. The fusion methods can be simple concatenation, weighted summation, and generating a unified feature representation through non-linear transformation. The formula is as follows: ; Among them, is the final fused feature of entity node (i.e., the multi-modal fusion data of traditional Chinese medicine), is the feature of entity node at the th layer, is the The features of a modal node at the layer, is the weight matrix, is the activation function.

[0091] 3) Inference logic: Information transmission ensures that the features of modal nodes can be transmitted layer by layer to entity nodes, and feature fusion generates a unified feature representation through different fusion methods (i.e., concatenation, weighted summation, and through non-linear transformation), thereby achieving deep fusion of multi-modal data.

[0092] (5) Output and optimization of the fusion result.

[0093] 1) Feature representation output: The finally generated feature vector (i.e., the multi-modal fusion data of traditional Chinese medicine) serves as the fusion feature of entity nodes and can reflect the information of multi-modal data simultaneously.

[0094] 2) Loss function: ; Among them, represents the number of prediction results, represents the true label, represents the predicted label (i.e., the value predicted by the model).

[0095] 3) Model optimization: Update the model parameters through backpropagation and optimization algorithms (such as the Adam optimization algorithm) to minimize the value of the loss function, thereby optimizing the fusion effect.

[0096] Through the above process, in this embodiment, the multi-modal data of traditional Chinese medicine is preprocessed through methods such as data preprocessing, normalization, and dimensionality reduction. Based on the graph neural network, data of different modalities is fused. This multi-modal traditional Chinese medicine data fusion representation scheme based on the graph neural network not only mines the potential key information of traditional Chinese medicine data but also significantly reduces the redundancy of traditional Chinese medicine data.

[0097] II. Knowledge graph representation scheme based on reinforcement learning.

[0098] Refer to Figure 5 , which is a schematic diagram of the construction process of the knowledge graph based on reinforcement learning. In order to effectively represent traditional Chinese medicine data, this embodiment proposes a knowledge graph representation scheme based on reinforcement learning. This scheme uses the knowledge graph to achieve visual representation of traditional Chinese medicine data and, with the help of the dynamic decision-making ability of reinforcement learning, optimizes the weight allocation and path inference efficiency of entity relationships in the knowledge graph.

[0099] 1. Introduction to the construction of the traditional Chinese medicine knowledge graph.

[0100] The construction of the Traditional Chinese Medicine (TCM) knowledge graph (i.e., the initial knowledge graph) is the first step of this solution. Its core idea is to transform TCM data into a visual graph structure. The specific process is as follows: (1) Data reception: Receive the multi-modal fusion data of TCM passed from the multi-modal data fusion module to this module.

[0101] (2) Definition of entity nodes: Take the core objects in the multi-modal data of TCM, such as Angelica sinensis and diarrhea, as entity nodes.

[0102] (3) Definition of entity node attributes: Take the multi-modal fusion data of TCM obtained from data reception as the attribute information of entity nodes.

[0103] (4) Definition of relationship edges: According to the types of entity nodes and the relationships between entity nodes, such as treatment relationship (Chinese medicine -> disease), compatibility relationship (Chinese medicine -> Chinese medicine), and taboo relationship (Chinese medicine <-> Chinese medicine), etc., as relationship edges and assign initial weights. The initial weight value is a pre-set value.

[0104] 2. Introduction to the dynamic weight adjustment based on Markov decision-making.

[0105] The dynamic weight adjustment based on Markov decision-making is the second step of this solution. Its core purpose is to dynamically optimize the weight distribution of entity relationships (such as "treatment", "compatibility", "taboo") in the TCM knowledge graph (i.e., the initial knowledge graph) through reinforcement learning strategies, so as to improve the practicality and safety of the recommended TCM paths and provide information and data basis for the subsequent two-layer divide-and-conquer architecture section. The specific process is as follows: (1) Definition of Markov state: The combination of multiple entity nodes is used as a composite state, reflecting the relationships of medicine pairs, prescriptions or other multi-dimensions.

[0106] (2) Definition of state transition rules: 1) Definition of transition probability calculation: The probability of transitioning from the composite state to the composite state is The formula is: ; 2) Definition of transition constraint rules: When the medicinal properties of the transferred entity nodes are in a taboo relationship, the transition probability is forced to be 0.

[0107] (3) Dynamic weight adjustment mechanism: 1) Positive feedback rule: When a certain path is verified to be effective, increase the weight of the relevant edge. The formula is: ; Among them, represents the initial weight value, represents the learning rate, Indicates the curative effect score.

[0108] 2) Reverse feedback rule: If a certain path triggers a taboo relationship or is invalid, reduce the weight of the relevant edge. The formula is: ; Where represents the initial weight value, represents the penalty coefficient, represents the severity of violation (e.g., 0.95 indicates a serious violation).

[0109] Through the above process, this embodiment obtains a traditional Chinese medicine knowledge graph dynamically adjusted by Markov decision, including entity nodes, the relationships between edges, and corresponding weights.

[0110] III. Auxiliary retrieval scheme based on a two-layer divide-and-conquer architecture.

[0111] To solve the problems that existing methods are difficult to uniformly process knowledge graphs containing multi-dimensional information such as taboo relationships, drug interactions, and patient symptoms, and face the problem of response delay caused by full-graph traversal analysis of ultra-large-scale graphs, which affects clinical real-time decision-making, this embodiment proposes a two-layer divide-and-conquer architecture to implement an intelligent processing scheme for complex graphs. Referring to Figure 6 , this scheme subdivides the original knowledge graph through two layers of architectures: graph deconstruction and agent interaction, so as to effectively process and utilize the conflicting information between graphs, and transmit the processed information to the large language model for auxiliary diagnosis and treatment.

[0112] This embodiment redefines the main graph of the traditional Chinese medicine knowledge graph as follows: Main graph structure: Defined as a weighted hypergraph G=(V, E, H), where the node set V includes medicinal materials, syndrome types, and meridian tropisms, etc. The binary edges E include compatibility taboos and synergistic effects, etc. The hyperedges H include multi-node relationships connecting compound prescriptions, syndrome types, and pathways. The node features are: each node has a feature vector , represents the dimension of the feature vector, includes attributes such as the four natures and five flavors, meridian tropisms, and toxicity.

[0113] 1. Introduction to the graph deconstruction layer.

[0114] The graph deconstruction layer is mainly used to cut the complex main graph into subgraphs suitable for different agents to process. Each subgraph contains the key information required for specific tasks and is transmitted to the subsequent agent interaction layer. The specific process is as follows: (1) Dynamic Feature Enhancement: The target agent is feature-enhanced by adopting a learnable feature projection matrix, and the original features are mapped to a task-adapted semantic space through linear transformation. This mechanism enhances the agent's perception ability of key attribute information through differential feature representation, thereby optimizing the subgraph segmentation effect based on feature correlation. Its mathematical expression is: ; Among them, represents the node feature vector, characterizing the inherent attributes such as the four natures and five flavors, and meridian tropism of traditional Chinese medicine materials; represents the agent 's exclusive trainable weight matrix to achieve dimensional to dimensional feature space transformation; represents the feature offset vector, which is used to adjust the projected feature distribution; represents the optimized feature vector (i.e., enhanced node feature vector) for the agent , enhancing the weights of task-related features.

[0115] Through the technical solution of this embodiment, different subgraphs in the main graph are associated with corresponding agents, effectively reducing redundant calculations and processing.

[0116] (2) Similarity Calculation and Graph Reconstruction.

[0117] In this embodiment, a dynamic graph structure is constructed based on the enhanced node feature vectors corresponding to the agent , and task-aware neighborhood discovery is achieved through attention-driven similarity measurement. Its calculation process is as follows: In the feature space of the agent , the cosine similarity is used to quantify the feature correlation strength between nodes : ; Among them, represents the th node, represents the th node, represents the feature vector of the th optimized node for the agent , represents the feature vector of the th optimized node for the agent , represents the cosine similarity calculation function.

[0118] Then, a sparse adjacency matrix is generated based on the similarity threshold to achieve adaptive reconstruction of the topological structure: ; Among them, represents the sparse adjacency matrix, represents the similarity threshold. Different agents correspond to different similarity thresholds, and the similarity threshold can be obtained by presetting the similarity threshold range or taking the similarity of the top 10% of the similarity rankings of the sparse adjacency matrix as the similarity threshold.

[0119] The technical solution of this embodiment realizes multi-dimensional structural cognition. The security agent constructs a taboo subgraph with high similarity connections in the projection space by strengthening the similarity of taboo features. The pharmacodynamic agent enhances the similarity of meridian entry features to form dense connections of nodes with similar meridian entries. The differentiation mechanism enables each agent to accurately capture the task-related topological patterns in the exclusive feature space.

[0120] (3) Multi-agent subgraph segmentation driven by spectral clustering.

[0121] After completing the adjacency relationship modeling of the multi-agent system, the scheme of this embodiment realizes subgraph segmentation through the spectral clustering algorithm. The specific implementation steps are as follows: First, construct an adjacency matrix based on the interaction topological structure of each agent , where the matrix element represents the connection strength between agent and the th node and the th node. Then construct the degree matrix D, which is a diagonal matrix, and its diagonal elements are defined as: ; Among them, represents the total number of nodes, is the sum of the connection degrees of the th node in the topological network corresponding to agent . Subsequently, construct the core analysis tool in graph theory, that is, the unnormalized Laplacian matrix, and its expression is: ; Among them, represents the unnormalized Laplacian matrix, represents the degree matrix, represents the sparse adjacency matrix.

[0122] In the feature analysis stage, the core steps are as follows: First, generate a special matrix (i.e., the non-normalized Laplacian matrix) based on the main graph structure, and extract multiple core eigenvectors from it. These core eigenvectors can best reflect the segmentation features of the network when combined. Then, use the K-means algorithm to group the data of these core eigenvectors, and each group corresponds to the management area of an agent. Finally, for the sub-network (i.e., the management area) responsible for each agent, its internal connection relationship completely maintains the original structure. This method can intelligently identify the tight groups in the network, ensure high connectivity within each divided sub-region, and maximize separation between different sub-regions.

[0123] 2. Introduction to the agent interaction layer.

[0124] This embodiment adopts a multi-agent collaborative architecture driven by a graph structure. By decomposing the main graph (i.e., the traditional Chinese medicine knowledge graph) into sub-graphs corresponding to agents and establishing a dynamic cooperation mechanism among agent interactions, accurate recommendation and safety verification of traditional Chinese medicine compound prescriptions are realized. The architecture specifically includes the following three parts: (1) The combined medicinal materials strategy of the efficacy agent is as follows: This agent acts on the efficacy sub-graph containing the relationship between symptoms and medicinal materials for treatment. By calculating the correlation degree of the medicinal materials related to the patient's symptoms in the efficacy sub-graph and using the agent to monitor and adjust the number of sovereign, ministerial, assistant, and envoy medicinal materials in real time to meet the rules of sovereign, ministerial, assistant, and envoy in traditional Chinese medicine compound prescriptions, the specific steps are as follows: First, this agent calculates the comprehensive efficacy score (correlation degree , that is, the average treatment intensity of this medicinal material for all the patient's symptoms) of each medicinal material from the knowledge sub-graph according to the patient's symptoms, that is: ; Among them, represents the comprehensive efficacy score, represents the set of the patient's symptoms, represents the symptoms of one disease in the set of the patient's symptoms, represents the medicinal materials used for the s symptom, represents the weight of the treatment edge formed by the symptom and the medicinal material, and this weight is a preset value (i.e., a value preset according to experience).

[0125] Then, in multiple rounds of iterative adjustment (up to 10 rounds of adjustment), the initial compound prescription in the initial iteration includes 8 medicinal materials. According to the comprehensive efficacy score, it is ensured that the compound prescription contains at least 2 highly effective core medicinal materials (highly effective core medicinal materials are those with a comprehensive efficacy score greater than or equal to 0.8 (i.e., the first preset value)) preferentially. For example, if "Astragalus membranaceus" has a strong curative effect on the patient's qi deficiency symptoms, it will be preferentially selected.

[0126] Then, inefficient medicinal materials (i.e., medicinal materials with a comprehensive efficacy score less than 0.5 (i.e., the second preset value)) are eliminated. For example, if "Licorice" has a weak effect on the current symptoms, it will be replaced. Subsequently, the number of medicinal materials will be supplemented to 8 (according to the limitation that the number of medicinal materials in traditional Chinese medicine compound prescriptions cannot be too large, the compound prescription in this embodiment includes 8 medicinal materials). Supplementary medicinal materials are added in descending order of score. For example, "Poria" may be selected because it takes into account promoting diuresis and strengthening the spleen. Finally, if the change in the composition of the prescription is less than 10% (i.e., the third preset value) in two consecutive rounds, or the fluctuation of the total efficacy score of all medicinal materials in the compound prescription is less than 5% (i.e., the fourth preset value), the plan is considered stable, and the optimization is stopped to obtain the first compound prescription. The total efficacy score is the sum of the comprehensive efficacy scores of all medicinal materials in the compound prescription. The thresholds here can be dynamically adjusted. Currently, the settings for high or medium and low thresholds are considered under normal circumstances.

[0127] It should be noted that the compound prescription in this embodiment may have only 8 medicinal materials, but it is not limited to only 8 medicinal materials. The number of medicinal materials in the compound prescription can be modified according to the actual situation, and no specific limitation is made in this embodiment.

[0128] (2) The safety intelligent agent's strategy for reviewing the safety of medicinal materials is as follows: This intelligent agent conducts a safety check on the first compound prescription, calculates the conflicts between medicinal materials themselves and between medicinal materials according to attributes such as the taboos between medicinal materials and the toxicity of the medicinal materials themselves, and eliminates the conflicting medicinal materials. Finally, a safe compound prescription is obtained. The specific steps are as follows: This intelligent agent acts on the safety sub-graph deconstructed from the traditional Chinese medicine knowledge graph , which contains the set of medicinal material entity nodes and the set of edges of compatibility taboos (such as classic taboo rules like "the eighteen incompatible medicinals", etc.). This embodiment realizes the optimization of the compound prescription through a dual safety verification mechanism. Its core process is: first, control the toxicity intensity, second, perform the resolution of compatibility conflicts, and finally output the second compound prescription that meets the safety standards of the Chinese Pharmacopoeia.

[0129] Specifically, in stage 1, two-level filtering is implemented through the calculation of toxicity attributes: (1) When the single-drug toxicity (the toxicity of each medicinal material recorded in the Chinese Pharmacopoeia) is greater than or equal to the first safety threshold, directly remove this highly toxic medicinal material (such as Aconite Root and Strychnos Seeds); (2) Calculate the total toxicity and the toxicity efficiency ratio. If the total toxicity is greater than the second safety threshold, remove the low-efficiency and highly toxic medicinal materials in the order of the toxicity efficiency ratio. It can be to remove the medicinal material with the lowest toxicity efficiency ratio, or to remove multiple medicinal materials with a toxicity efficiency ratio lower than the preset value. The specific number of medicinal materials to be removed can be changed according to the actual situation, and no specific limitation is made in this embodiment. The formula for calculating the toxicity efficiency ratio is: ; Among them, represents the toxicity efficiency ratio, Indicates the contribution degree of the medicinal material to the therapeutic effect (calculated by the PageRank value of the medicinal material on the efficacy sub - graph). Indicates the total toxicity.

[0130] In stage 2, conflict detection is carried out on the compatibility sub - graph. For the pairs of medicinal materials with edges of incompatibility relationships , calculate the topological fitness (the topological fitness reflects the compatibility of the medicinal material with the current compound prescription): ; Among them, Indicates the topological fitness, Indicates the set of neighbor nodes of it in the efficacy sub - graph, Indicates a compliant compound prescription.

[0131] Select the medicinal materials with lower fitness for removal. By adopting a dynamic iterative architecture, when there is no change in the compound prescription after two consecutive iterative adjustments, the process terminates, and the second compound prescription is obtained. This embodiment not only ensures sufficient optimization but also avoids infinite loops.

[0132] (3) Multi - agent collaboration: This embodiment proposes a dual - agent iterative optimization framework based on a knowledge graph for generating traditional Chinese medicine prescriptions that take into account both efficacy and safety. Its core mechanism realizes dynamic optimization through the alternating execution of combination generation and safety verification. The specific process is as follows: This embodiment realizes the dynamic optimization of the compound prescription through multi - agent iterative collaboration: First, the efficacy agent optimizes the initial compound prescription to generate the first compound prescription based on the patient's symptom set, the medicinal material sub - graph, and the safety sub - graph. Its core is to screen the compatibility of medicinal materials according to traditional Chinese medicine theories (such as monarch, minister, assistant, and courier, nature, flavor, and meridian tropism); Subsequently, the safety agent conducts contraindication verification and conflict resolution on the first compound prescription, and uses the drug interaction and toxicity rules in the graph to eliminate high - risk combinations and outputs the second compound prescription; If the second compound prescription does not change after two consecutive iterations, it is considered to reach a stable state, the loop is terminated, and the final compound prescription (i.e., the optimal compound prescription) is output.

[0133] 3. Introduction to large language model - assisted diagnosis and treatment.

[0134] The large language model screens out the "monarch, minister, assistant, and courier" compatibility based on a divide - and - conquer architecture, and combines the traditional Chinese medicine knowledge graph to analyze the nature, flavor, and meridian tropism of medicinal materials and their synergistic laws. Based on the "efficacy of the monarch drug" as the benchmark, the dosage is dynamically allocated according to the "monarch and minister" levels, and the assistant and courier drugs are fine - tuned. The large language model is fine - tuned using multiple traditional Chinese medicine ancient books such as the Chinese Pharmacopoeia to obtain professional knowledge in the field. The final compound prescription is input into the fine - tuned large language model, and the large language model generates a complete compound prescription (i.e., the target compound prescription) containing accurate dosage, decocting and taking methods, and treatment courses to ensure that the target compound prescription conforms to the prescription principle of "orderly primary and secondary, appropriate tonifying and purging".

[0135] Compared with the prior art, the technical solution of this embodiment has the following advantages: The technical solution of this embodiment is based on a multi-modal traditional Chinese medicine data fusion using a graph neural network, a knowledge graph representation using reinforcement learning, and an auxiliary retrieval scheme with a two-layer divide-and-conquer architecture. By means of multi-modal data fusion, dynamically adjusting the traditional Chinese medicine knowledge graph using reinforcement learning, auxiliary retrieval with a two-layer divide-and-conquer architecture, and generating target compound prescriptions using a large language model, the problems of large computational overhead, high complexity, and difficulty in applying to clinical diagnosis in traditional methods are successfully solved. The technical solution of this embodiment not only improves the accuracy and efficiency of compound prescription recommendation, significantly reduces the dependence on a large amount of computing resources, but also successfully mines the potential information in traditional Chinese medicine data. In addition, the interpretability of the model is improved through the representation structure of the knowledge graph, making the content of traditional Chinese medicine data more transparent and intuitive. The technical solution of this embodiment is easy to understand and analyze and has broad application prospects.

[0136] Refer to Figure 7 , the embodiment of this application also provides a traditional Chinese medicine recommendation system based on multi-modal fusion. The system includes a data acquisition unit 100, a data processing unit 200, a heterogeneous graph construction unit 300, a data fusion unit 400, a graph construction unit 500, a graph segmentation unit 600, a collaborative optimization unit 700, and a compound prescription determination unit 800, where: The data acquisition unit 100 is used to acquire traditional Chinese medicine multi-modal data including text data, image data, audio data, and video data; The data processing unit 200 is used to preprocess the traditional Chinese medicine multi-modal data to obtain preprocessed traditional Chinese medicine multi-modal data; The heterogeneous graph construction unit 300 is used to construct a heterogeneous graph according to the core representative objects and the preprocessed traditional Chinese medicine multi-modal data, where the core representative objects include the core medicinal materials in the traditional Chinese medicine multi-modal data; The data fusion unit 400 is used to input the heterogeneous graph into a graph neural network, and fuse the preprocessed traditional Chinese medicine multi-modal data through the graph neural network to obtain traditional Chinese medicine multi-modal fusion data; The graph construction unit 500 is used to construct a knowledge graph based on the traditional Chinese medicine multi-modal fusion data to obtain a traditional Chinese medicine knowledge graph; The graph segmentation unit 600 is used to divide the traditional Chinese medicine knowledge graph into subgraphs processed by multiple agents according to efficacy and medication safety to obtain a medicinal material subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent, where the multiple agents include two types: an efficacy agent and a safety agent. The efficacy agent includes a combined medicinal material strategy, and the safety agent includes a strategy for reviewing the safety of medicinal materials; The collaborative optimization unit 700 is used to perform collaborative iterative optimization selection of medicinal materials based on the medicinal material sub-graph and the safety sub-graph, by combining the medicinal material strategy and the strategy for reviewing the safety of medicinal materials, so as to obtain the optimal compound prescription for traditional Chinese medicine recommendation; The compound prescription determination unit 800 is used to input the optimal compound prescription into the fine-tuned large language model to determine the target compound prescription for traditional Chinese medicine recommendation.

[0137] It should be noted that since the traditional Chinese medicine recommendation system based on multi-modal fusion in this embodiment and the above-mentioned traditional Chinese medicine recommendation method based on multi-modal fusion are based on the same inventive concept, the corresponding content in the method embodiment is equally applicable to the system embodiment of this application, and will not be elaborated here.

[0138] This application embodiment also provides an electronic device, including: at least one control processor and a memory for communicatively connecting with at least one control processor.

[0139] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0140] The non-transitory software programs and instructions required to implement the above-mentioned traditional Chinese medicine recommendation method based on multi-modal fusion in the above embodiment are stored in the memory, and when executed by the processor, they execute the above-mentioned traditional Chinese medicine recommendation method based on multi-modal fusion in the above embodiment. For example, they execute the Figure 1 method steps S100 to step S800 described above.

[0141] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] This application embodiment also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more control processors, the one or more control processors can execute the above-mentioned traditional Chinese medicine recommendation method based on multi-modal fusion in the method embodiment. For example, they execute the Figure 1The functions of method steps S100 to S800 in

[0143] Those of ordinary skill in the art will appreciate that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, a communication medium typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0144] The above is a specific description of the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the embodiments of the present application, and these equivalent deformations or substitutions are all included within the scope defined by the claims of the embodiments of the present application.

Claims

1. A traditional Chinese medicine recommendation method based on multimodal fusion, characterized in that, The method includes: Obtain multi-modal traditional Chinese medicine data including text data, image data, audio data, and video data; Preprocess the multi-modal traditional Chinese medicine data to obtain preprocessed multi-modal traditional Chinese medicine data; Construct a heterogeneous graph based on the core representative objects and the preprocessed multi-modal traditional Chinese medicine data, where the core representative objects include the core medicinal materials in the multi-modal traditional Chinese medicine data; Input the heterogeneous graph into a graph neural network, and fuse the preprocessed multi-modal traditional Chinese medicine data through the graph neural network to obtain multi-modal traditional Chinese medicine fusion data; Construct a knowledge graph based on the multi-modal traditional Chinese medicine fusion data to obtain a traditional Chinese medicine knowledge graph; Segment the traditional Chinese medicine knowledge graph into subgraphs processed by multiple agents according to efficacy and medication safety to obtain a medicinal material subgraph associated with an efficacy agent and a safety subgraph associated with a safety agent, where the multiple agents include two types: an efficacy agent and a safety agent, the efficacy agent includes a combined medicinal material strategy, and the safety agent includes a strategy for reviewing the safety of medicinal materials; Based on the medicinal material subgraph and the safety subgraph, select medicinal materials through collaborative iterative optimization using the combined medicinal material strategy and the strategy for reviewing the safety of medicinal materials to obtain the optimal compound prescription recommended by traditional Chinese medicine; Input the optimal compound prescription into a fine-tuned large language model to determine the target compound prescription recommended by traditional Chinese medicine.

2. The traditional Chinese medicine recommendation method based on multimodal fusion according to claim 1, wherein The preprocessing of the multi-modal traditional Chinese medicine data to obtain preprocessed multi-modal traditional Chinese medicine data includes: Preprocess the text data in the multi-modal traditional Chinese medicine data using word embedding to obtain a first feature vector; Preprocess the image data in the multi-modal traditional Chinese medicine data using a convolutional neural network to obtain a second feature vector; Preprocess the audio data in the multi-modal traditional Chinese medicine data using a deep learning model to obtain a third feature vector; Preprocess the video data in the multi-modal traditional Chinese medicine data using a convolutional neural network to obtain a fourth feature vector; Perform normalization and dimensionality reduction processing on the first feature vector, the second feature vector, the third feature vector, and the fourth feature vector to obtain preprocessed multi-modal traditional Chinese medicine data.

3. The traditional Chinese medicine recommendation method based on multimodal fusion according to claim 1, wherein The constructing of the knowledge graph based on the multi-modal traditional Chinese medicine fusion data to obtain a traditional Chinese medicine knowledge graph includes: Select the core objects in the multi-modal traditional Chinese medicine data as entity nodes, where the core objects include the medicinal materials and symptoms in the multi-modal traditional Chinese medicine data; Use the multi-modal traditional Chinese medicine fusion data as the attribute information of the entity nodes; Construct relationship edges according to the relationships between the entity nodes and preset initial weights for the relationship edges; Construct an initial knowledge graph based on the entity nodes, the attribute information, and the relationship edges; Adjust the initial weights of the relationship edges in the initial knowledge graph using Markov decision-making to obtain optimized weights; Replace the initial weights of the relationship edges in the initial knowledge graph with the optimized weights to construct a traditional Chinese medicine knowledge graph.

4. The traditional Chinese medicine recommendation method based on multimodal fusion according to claim 1, wherein Segmenting the traditional Chinese medicine knowledge graph into subgraphs processed by multiple agents according to drug efficacy and medication safety to obtain a medicinal material subgraph associated with a drug efficacy agent and a safety subgraph associated with a safety agent, including: Constructing the traditional Chinese medicine knowledge graph into a main graph structure including nodes, binary edges, hyperedges, and node feature vectors; Letting multiple agents learn the node feature vectors in the main graph structure, and using a feature projection matrix to enhance the node feature vectors during the learning process to obtain enhanced node feature vectors corresponding to each agent; Calculating the similarity between nodes according to the enhanced node feature vectors corresponding to each agent, and constructing a sparse adjacency matrix corresponding to each agent based on the similarity; Constructing a degree matrix according to the sparse adjacency matrix corresponding to each agent; Subtracting the degree matrix from the sparse adjacency matrix to obtain an unnormalized Laplacian matrix; Clustering the eigenvectors in the unnormalized Laplacian matrix to obtain multiple groups of clustering results; Taking each group of clustering results as a subgraph processed by an agent to obtain a medicinal material subgraph associated with a drug efficacy agent and a safety subgraph associated with a safety agent.

5. The traditional Chinese medicine recommendation method based on multimodal fusion according to claim 1, wherein Based on the medicinal material subgraph and the safety subgraph, synergistically and iteratively optimizing the selection of medicinal materials through the combined medicinal material strategy and the medicinal material safety review strategy to obtain the optimal compound prescription recommended by traditional Chinese medicine, including: Using the combined medicinal material strategy to select medicinal materials from the medicinal material subgraph for combination to obtain a first compound prescription; Based on the safety subgraph, using the medicinal material safety review strategy to conduct a safety review of the medicinal materials in the first compound prescription to obtain a second compound prescription; Synergistically and iteratively optimizing the combined medicinal material strategy and the medicinal material safety review strategy until the second compound prescriptions in two consecutive synergistic iterative optimizations are the same to obtain the optimal compound prescription recommended by traditional Chinese medicine.

6. The traditional Chinese medicine recommendation method based on multimodal fusion according to claim 5, wherein, The medicinal material subgraph includes the relationship between symptoms and the treatment of medicinal materials. Using the combined medicinal material strategy to select medicinal materials from the medicinal material subgraph for combination to obtain a first compound prescription, including: According to the patient's symptoms, calculating the comprehensive efficacy score of each medicinal material in the medicinal material subgraph for the symptoms; Performing iterative adjustment according to the combined medicinal material strategy, including: selecting a preset number of medicinal materials from high to low according to the comprehensive efficacy score to obtain an initial compound prescription; Retaining the medicinal materials in the initial compound prescription whose comprehensive efficacy score is greater than or equal to a first preset value; After removing the medicinal materials in the initial compound prescription whose comprehensive efficacy score is less than a second preset value, and supplementing to the preset number of medicinal materials from high to low according to the comprehensive efficacy score; Until the change in the medicinal material components in the compound prescriptions obtained by two consecutive rounds of iterative adjustment of the combined medicinal material strategy is less than a third preset value, or the total efficacy score fluctuation of all medicinal materials in the compound prescription is less than a fourth preset value, to obtain a first compound prescription.

7. The traditional Chinese medicine recommendation method based on multimodal fusion according to claim 5, wherein The safety subgraph includes medicinal material entity nodes and compatibility taboo relationship edges. Based on the safety subgraph, using the medicinal material safety review strategy to conduct a safety review of the medicinal materials in the first compound prescription to obtain a second compound prescription, including: Iteratively adjust according to the reviewed traditional Chinese medicine (TCM) material safety strategy, including: obtaining the single-herb toxicity of each herb, and removing the herbs in the first compound with a single-herb toxicity greater than or equal to the first safety threshold; Calculating the total toxicity of all herbs in the first compound, calculating the contribution degree of the treatment effect of the herbs, and calculating the toxicity efficiency ratio of each herb according to the contribution degree of the treatment effect and the single-herb toxicity; If the total toxicity is greater than the second safety threshold, remove the herbs with a low toxicity efficiency ratio; If there is a compatibility taboo relationship edge in the first compound, obtain the herb pair according to the two herb entity nodes corresponding to the compatibility taboo relationship edge; Calculating the topological fitness of each herb in the herb pair, and removing the herb with a low topological fitness in the herb pair; Until the herbs in the compound in two consecutive rounds of iterative adjustment of the reviewed TCM material safety strategy remain unchanged, a second compound is obtained.

8. A traditional Chinese medicine recommendation system based on multimodal fusion, characterized in that, The system includes: A data acquisition unit for acquiring multi-modal TCM data including text data, image data, audio data, and video data; A data processing unit for preprocessing the multi-modal TCM data to obtain preprocessed multi-modal TCM data; A heterogeneous graph construction unit for constructing a heterogeneous graph according to the core representative object and the preprocessed multi-modal TCM data, where the core representative object includes the core herbs in the multi-modal TCM data; A data fusion unit for inputting the heterogeneous graph into a graph neural network, and fusing the preprocessed multi-modal TCM data through the graph neural network to obtain multi-modal TCM fusion data; A knowledge graph construction unit for constructing a knowledge graph based on the multi-modal TCM fusion data to obtain a TCM knowledge graph; A knowledge graph segmentation unit for segmenting the TCM knowledge graph into subgraphs processed by multiple agents according to the efficacy and medication safety, to obtain a herb subgraph associated with the efficacy agent and a safety subgraph associated with the safety agent, where the multiple agents include two types: an efficacy agent and a safety agent, the efficacy agent includes a combined herb strategy, and the safety agent includes a reviewed TCM material safety strategy; A collaborative optimization unit for collaboratively and iteratively optimizing the selection of herbs based on the herb subgraph and the safety subgraph through the combined herb strategy and the reviewed TCM material safety strategy to obtain the optimal compound recommended by TCM; A compound determination unit for inputting the optimal compound into a fine-tuned large language model to determine the target compound recommended by TCM.

9. An electronic device, characterized in that, It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the multi-modal fusion-based TCM recommendation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to execute the multi-modal fusion-based TCM recommendation method according to any one of claims 1 to 7.

Images