A method and system for fine-tuning and light inference of a local traditional Chinese medicine model based on a fusion graph vector embedding

Abstract

The present application relates to the field of artificial intelligence and traditional Chinese medicine information processing technology, especially to a local traditional Chinese medicine large model fine-tuning and light reasoning method and system fusing graph vector embedding, the method comprising: constructing a hierarchical traditional Chinese medicine knowledge graph containing general knowledge and local characteristic knowledge, and generating graph vector embedding; fine-tuning a pre-trained large language model by using a graph vector fusion attention mechanism, injecting the graph vector embedding into self-attention calculation, making the model fuse structured knowledge, and obtaining a teacher model; and migrating the knowledge of the teacher model to a light student model by knowledge distillation by using a composite loss function containing graph structure consistency loss. The present application deeply fuses structured knowledge and text semantics, improves the knowledge extraction accuracy and reasoning ability of local characteristic traditional Chinese medicine literature, and generates an efficient light-weight model suitable for grassroots environment.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and traditional Chinese medicine information processing technology, and in particular to a method and system for fine-tuning and lightweight inference of a local TCM large model that integrates graph vector embedding. Background Technology

[0002] Traditional Chinese medicine (TCM) is a treasure of the Chinese nation, its vast collection of literature containing thousands of years of accumulated clinical experience and profound theoretical thought. However, these documents are mostly written in classical Chinese and semi-structured medical records, characterized by complex terminology, strong implicit knowledge, and distinct regional features. For example, Henan Province, as an important birthplace of TCM culture, contains a wealth of regionally specific knowledge elements and unique knowledge connections in its distinctive literature (such as the interpretation of the Nanyang School of medicine in the *Treatise on Febrile and Miscellaneous Diseases*, and records of the application of authentic "Huai medicine") (e.g., "a certain prescription is adapted from a certain classic prescription"). How to efficiently mine, organize, and apply these valuable knowledge resources using artificial intelligence technology has become an important issue in the informatization and intelligent development of TCM.

[0003] In recent years, artificial intelligence technologies, represented by large language models (LLMs), have made breakthrough progress in natural language processing tasks, bringing new opportunities for the intelligent processing of traditional Chinese medicine (TCM) knowledge. However, general-purpose large language models face the following core challenges when processing highly specialized and regionalized TCM texts: First, there is the issue of knowledge gaps and factual fallacies. The pre-training corpus of the general-purpose language model mainly consists of internet texts from general domains, with a serious lack of coverage of specialized and regionally specific TCM knowledge. When the model processes local TCM literature, it is prone to "knowledge illusion," that is, generating content that does not conform to basic TCM theories or local practices, which is unacceptable in high-risk fields such as healthcare.

[0004] Second, there is a lack of understanding of complex knowledge relationships. The knowledge relationships in traditional Chinese medicine literature are highly dynamic and complex, such as "modification of prescriptions according to symptoms," "transmission and meridian tropism of diseases," and "modification and evolution of prescriptions." These relationships are often implicit in the text and are difficult to model using simple linear contexts. Traditional large-scale language models mainly rely on the statistical patterns of text sequences for learning, which makes it difficult to accurately capture and infer these deep-seated, structured knowledge connections.

[0005] Third, model deployment and computing power adaptation are challenging. Current mainstream large language models have a massive number of parameters, and direct full-parameter fine-tuning requires exorbitant computing resources and storage costs. However, for practical applications such as primary healthcare institutions and mobile diagnostic devices, computing resources are generally limited, necessitating lightweight models that can operate efficiently in low-resource environments. How to compress model size while maximizing the preservation of the model's understanding and reasoning capabilities for professional knowledge is a key technical challenge.

[0006] To address the aforementioned issues, some exploratory solutions have been explored in existing technologies. A common approach is to combine knowledge graphs with language models, providing relevant knowledge at the model input through Retrieval-Enhanced Generation (RAG) technology. While this method can alleviate the knowledge blind spot problem to some extent, the knowledge integration remains at a superficial level. The structured information in the knowledge graph is only utilized at the input layer; the model does not truly "understand" and "internalize" the structural features of this knowledge internally, resulting in limited improvement in the ability to reason about complex relationships.

[0007] Another technical approach is to use parameter-efficient fine-tuning (PEFT) methods, such as low-rank adaptation (LoRA), to reduce the computational cost of model fine-tuning. These methods mainly address the training efficiency issue, but they do not address how to introduce external structured knowledge into the model's internal learning process, and therefore cannot fundamentally improve the model's depth of understanding of domain-specific knowledge.

[0008] In terms of model lightweighting, traditional knowledge distillation techniques primarily focus on making the student model mimic the output probability distribution of the teacher model, transferring knowledge through soft-label loss. However, this distillation approach neglects the structured reasoning capabilities inherent in the teacher model—especially when the teacher model has incorporated knowledge graph information, this capability is precisely key to improving the model's professional performance. Therefore, the student model compressed using conventional distillation methods often loses important structured knowledge, leading to a significant decline in reasoning ability.

[0009] In summary, there is an urgent need for a technical solution that can deeply integrate local TCM knowledge graphs into large-scale language models, effectively improving the model's understanding and reasoning accuracy of professional texts, and efficiently generating lightweight models suitable for resource-constrained environments. Summary of the Invention

[0010] In order to overcome the above-mentioned defects of the prior art, the present invention provides a method and system for fine-tuning and lightweight inference of a local TCM large model that integrates graph vector embedding, so as to solve the problems existing in the background art.

[0011] This invention provides the following technical solution: a method for fine-tuning and lightweight inference of a large-scale local TCM model integrating graph vector embedding, comprising the following steps: The steps for constructing a hierarchical TCM knowledge graph and generating graph vector embeddings are as follows: Based on the general TCM knowledge system and TCM literature of a specific region, a hierarchical knowledge graph containing a general knowledge layer and a local characteristic knowledge layer is constructed, and the hierarchical knowledge graph is trained using a knowledge graph representation learning algorithm to obtain the graph vector embeddings of each knowledge entity and knowledge relationship therein. The steps for fine-tuning a pre-trained large-scale language model through knowledge fusion are as follows: A graph vector fusion attention mechanism is used to fine-tune the pre-trained large-scale language model. The graph vectors are embedded and injected into the self-attention calculation module inside the model, so that the model integrates structured graph knowledge when processing input text, and a knowledge-enhanced teacher model is trained. The steps for knowledge-guided lightweight distillation are as follows: Design a composite loss function that includes graph structure consistency loss, and use knowledge distillation technology to transfer the knowledge of the teacher model to a student model with fewer parameters, thereby obtaining a lightweight model that retains the key knowledge reasoning ability.

[0012] Furthermore, the steps for constructing a hierarchical TCM knowledge graph include: constructing a general knowledge layer based on the national standard for TCM language system, containing core concepts of symptoms, prescriptions, herbs, and treatments and their interrelationships; mining and defining knowledge elements with regional characteristics from TCM literature in specific regions, including local entities and special relationships, as a local characteristic knowledge layer; and integrating the general knowledge layer with the local characteristic knowledge layer to form a unified hierarchical knowledge graph.

[0013] Furthermore, the fine-tuning step of the graph vector fusion attention mechanism specifically includes: performing entity recognition on the input text to determine the lexical units or lexical unit intervals in the text that correspond to the knowledge entities in the hierarchical knowledge graph; for each lexical unit corresponding to the identified knowledge entity, fusing its hidden layer representation in the model with the corresponding entity vector retrieved from the graph vector embedding to generate a knowledge-enhanced lexical representation; using the knowledge-enhanced lexical representation to calculate the query, key, and value matrix in the self-attention mechanism, and completing the attention score calculation and weighted summation operation.

[0014] Furthermore, the knowledge-enhanced lexical representation The calculation formula is: in, For the model The original hidden layer representation of each word element, For the graph vector embedding of the knowledge entity corresponding to this word element, and The projection matrix is ​​learnable. For bias terms, This is a layer normalization operation.

[0015] Furthermore, in the step of performing knowledge-guided lightweight distillation, the composite loss function The expression is: in, The standard cross-entropy loss of the student model on the real labels; The KL divergence loss is used to compare the soft labels output by the student model and the teacher model. This represents the loss of consistency in the spectral structure. These are the weighting coefficients for each type of loss.

[0016] Furthermore, the loss of consistency of the map structure The calculation method is as follows: for each knowledge triple (head entity) that appears in the training data ,relation Tail entity Extract the vector representations of the head and tail entities from the hidden states of the student model. and Define a relation that simulates a student model representation space. Transformation operations This operation is designed to represent the head entity. Transformed into a predicted tail entity representation; the graph structure consistency loss Defined as the predicted tail entity representation and the true tail entity representation The distance between them is given by the formula: in, The set of all knowledge triples in the training set. This represents the square of the L2 norm.

[0017] A local TCM large-scale model fine-tuning and lightweight inference system integrating graph vector embedding includes: a knowledge graph construction module, used to construct a hierarchical knowledge graph containing a general knowledge layer and a local characteristic knowledge layer based on a general TCM knowledge system and TCM literature of a specific region, and to train the hierarchical knowledge graph using a knowledge graph representation learning algorithm to obtain the graph vector embedding of each knowledge entity and knowledge relationship therein; The model fine-tuning module is used to fine-tune the pre-trained large language model using a graph vector fusion attention mechanism. The graph vectors are embedded into the self-attention calculation module inside the model, so that the model integrates structured graph knowledge when processing input text, and trains a knowledge-enhanced teacher model. The model distillation module is used to design a composite loss function that includes graph structure consistency loss. It employs knowledge distillation technology to transfer the knowledge of the teacher model to the student model with fewer parameters, thereby obtaining a lightweight model that retains the ability to reason about knowledge.

[0018] Furthermore, the model fine-tuning module is specifically used for: Entity recognition is performed on the input text to determine the lexical units or lexical units in the text that correspond to the knowledge entities in the hierarchical knowledge graph. For each lexical unit corresponding to the identified knowledge entity, its hidden layer representation in the model is fused with the corresponding entity vector retrieved from the graph vector embedding to generate a knowledge-enhanced lexical representation. The query, key, and value matrix in the self-attention mechanism is calculated using the knowledge-enhanced lexical representation, and attention score calculation and weighted summation operations are performed.

[0019] An electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of the preceding descriptions.

[0020] A computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method as described in any one of the preceding descriptions.

[0021] The technical effects and advantages of this invention are as follows: Compared with existing technologies, this invention embeds a knowledge graph into the self-attention computation process of a large language model through a graph vector fusion attention mechanism. This enables the model to utilize structured knowledge in text encoding, resulting in stronger modeling capabilities for knowledge relationships in professional domain texts compared to retrieval enhancement or shallow splicing methods. This reduces knowledge bias when general models process vertical domain texts. By constructing a hierarchical knowledge graph containing a general knowledge layer and a local characteristic knowledge layer, this invention is compatible with both common and regional knowledge in traditional Chinese medicine, making it suitable for processing TCM literature with local characteristics. In the model lightweighting stage, by introducing graph structure consistency loss to constrain the hidden layer representation space of the student model, the compressed small-scale model can retain the structured information of the knowledge graph. Compared with distillation methods that only simulate output distribution, it can still maintain a certain level of knowledge reasoning ability even with fewer parameters. The resulting lightweight model has fewer parameters and can be deployed in environments with limited computing power, such as primary healthcare institutions or mobile terminals. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall process of a method for fine-tuning and lightweight inference of a local TCM big data model with fusion graph vector embedding, provided in an embodiment of the present invention.

[0023] Figure 2 This is a detailed structural diagram of the graph vector fusion attention mechanism in an embodiment of the present invention.

[0024] Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0025] The following is for illustrative purposes only and is not intended to limit the scope of protection of the invention.

[0026] This invention provides a method for fine-tuning and lightweight inference of a large-scale local TCM model that integrates graph vector embedding, comprising the following steps: The steps for constructing a hierarchical TCM knowledge graph and generating graph vector embeddings are as follows: Based on the general TCM knowledge system and TCM literature from a specific region, a hierarchical knowledge graph containing a general knowledge layer and a local characteristic knowledge layer is constructed. The hierarchical knowledge graph is then trained using a knowledge graph representation learning algorithm to obtain the graph vector embeddings of each knowledge entity and knowledge relationship. The steps for fine-tuning a pre-trained large-scale language model through knowledge fusion are as follows: A graph vector fusion attention mechanism is used to fine-tune the pre-trained large-scale language model. The graph vectors are embedded and injected into the self-attention calculation module inside the model, so that the model integrates structured graph knowledge when processing input text, and a knowledge-enhanced teacher model is trained. The steps for knowledge-guided lightweight distillation are as follows: Design a composite loss function that includes graph structure consistency loss, and use knowledge distillation technology to transfer the knowledge of the teacher model to a student model with fewer parameters, thereby obtaining a lightweight model that retains the key knowledge reasoning ability.

[0027] The steps for constructing a hierarchical TCM knowledge graph include: defining a general knowledge layer: based on standards such as the National TCM Language System (TCMLS), defining a general ontology covering core concepts such as symptoms, prescriptions, herbs, and treatment methods and their interrelationships; defining a local characteristic knowledge layer: mining and defining knowledge elements with regional characteristics from TCM literature in specific regions (such as medical records and records of authentic medicinal materials in Henan Province), such as entities like "Huai medicine," "Zhongjing's theories," and "Nanyang School of Medicine," as well as special relationships like "adaptation," as extensions of the general knowledge layer; and integrating the general knowledge layer and the local characteristic knowledge layer to form a unified hierarchical knowledge graph.

[0028] The fine-tuning steps of the graph vector fusion attention mechanism specifically include: performing entity recognition on the input text to determine the lexical units or lexical units in the text that correspond to the knowledge entities in the hierarchical knowledge graph; for each lexical unit corresponding to the identified knowledge entity, fusing its hidden layer representation in the model with the corresponding entity vector retrieved from the graph vector embedding to generate a knowledge-enhanced lexical representation; using the knowledge-enhanced lexical representation to calculate the query (Q), key (K), and value (V) matrices in the self-attention mechanism, and completing the subsequent attention score calculation and weighted summation operation.

[0029] The knowledge-enhanced lexical representation The calculation formula is: in, For the model The original hidden layer representation of each word element, For the graph vector embedding of the knowledge entity corresponding to this word element, and For a learnable projection matrix, For bias terms, This is a layer normalization operation.

[0030] In the step of performing knowledge-guided lightweight distillation, the composite loss function The expression is: in, The standard cross-entropy loss of the student model on the real labels; The KL divergence loss is used to compare the soft labels output by the student model and the teacher model. This represents the loss of consistency in the spectral structure. These are the weighting coefficients for each type of loss.

[0031] The loss of consistency of the map structure The calculation method is as follows: for each knowledge triple (head entity) that appears in the training data ,relation Tail entity Extract the vector representations of the head and tail entities from the hidden states of the student model. and Define a relation that simulates a student model representation space. Transformation operations This operation is designed to represent the head entity. Transformed into a predicted tail entity representation; the graph structure consistency loss Defined as the predicted tail entity representation and the true tail entity representation The distance between them is given by the formula: in, The set of all knowledge triples in the training set. This represents the square of the L2 norm.

[0032] A local TCM large-scale model fine-tuning and lightweight inference system integrating graph vector embedding includes: a knowledge graph construction module, used to construct a hierarchical knowledge graph containing a general knowledge layer and a local characteristic knowledge layer based on a general TCM knowledge system and TCM literature of a specific region, and to train the hierarchical knowledge graph using a knowledge graph representation learning algorithm to obtain the graph vector embedding of each knowledge entity and knowledge relationship therein; The model fine-tuning module is used to fine-tune the pre-trained large language model using a graph vector fusion attention mechanism. The graph vectors are embedded into the self-attention calculation module inside the model, so that the model integrates structured graph knowledge when processing input text, and trains a knowledge-enhanced teacher model. The model distillation module is used to design a composite loss function that includes graph structure consistency loss. It employs knowledge distillation technology to transfer the knowledge of the teacher model to the student model with fewer parameters, thereby obtaining a lightweight model that retains the ability to reason about knowledge.

[0033] The model fine-tuning module is specifically used for: performing entity recognition on the input text to determine the lexical units or lexical units in the text that correspond to the knowledge entities in the hierarchical knowledge graph; for each lexical unit corresponding to the identified knowledge entity, fusing its hidden layer representation in the model with the corresponding entity vector retrieved from the graph vector embedding to generate a knowledge-enhanced lexical representation; using the knowledge-enhanced lexical representation to calculate the query (Q), key (K), and value (V) matrices in the self-attention mechanism, and completing the subsequent attention score calculation and weighted summation operation.

[0034] An electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of the preceding descriptions.

[0035] A computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method as described in any one of the preceding descriptions.

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. Example

[0037] This embodiment proposes an innovative methodological framework called "Hierarchical Knowledge-Injected Fine-Tuning and Inference" (HKI-FTI), aiming to address the limitations of general-purpose large-scale language models when processing locally specific TCM literature. The overall process of this framework is as follows: Figure 1 As shown, it mainly includes three core stages: 100) hierarchical construction and embedding of TCM knowledge graph, 200) model fine-tuning based on graph vector fusion attention, and 300) knowledge-guided lightweight distillation.

[0038] Phase 1: Construction and Embedding of Hierarchical TCM Knowledge Graph (100) The goal of this stage is to build a structured knowledge base that can comprehensively and accurately describe local TCM knowledge, and to vectorize it to provide a foundation for subsequent model fusion.

[0039] 1.1 Hierarchical Knowledge Graph Construction (110) In order to balance the universality and local specificity of TCM knowledge, we construct a two-layer knowledge graph. **General Knowledge Layer (L1):** This layer is based on authoritative national standards, such as the Traditional Chinese Medicine Language System (TCMLS), to construct a general TCM ontology. This layer defines the basic concept categories in the field of TCM (e.g., 96 categories such as Chinese herbal medicines, prescriptions, disease names, syndromes, and treatment methods) and the universal relationships between them (e.g., composition (is_composed_of), treatments (treats), efficacy (has_effect)). **Local Characteristic Knowledge Layer (L2):** This layer focuses on extracting unique knowledge from TCM literature of specific regions (this example uses Henan Province as an example) as an extension of the L1 layer. Specific work includes: **Characteristic Entity Mining:** Identifying and labeling TCM concepts unique to Henan, such as the "Four Great Huai Medicines" (Huai yam, Huai rehmannia, Huai achyranthes, and Huai chrysanthemum), local medical schools such as the "Nanyang School" and the "Heluo School," and medical terminology specific to different physicians. Specialized Relationship Definitions: This section defines and labels unique relationships that reflect local medical philosophies. For example, "is_derived_from" represents the evolutionary relationship between a local prescription and a classic formula, while "is_compatible_with_in" describes the drug compatibility rules of a specific medical school. Ultimately, by aligning the entities and relationships of the L2 layer and integrating them into the L1 layer framework, a unified hierarchical knowledge graph is formed. .

[0040] 1.2 Knowledge Graph Embedding (120) To enable the model to compute and process knowledge in the graph, it needs to be vectorized. We use a knowledge graph representation learning algorithm to embed the knowledge graph into vectors. Each entity in and relationships Mapping to a low-dimensional continuous vector space yields entity embeddings. and relational embedding ,in This refers to the embedding dimension. In this embodiment, a model capable of handling complex relationships (such as symmetric / antisymmetric, transitive) can be selected, such as RotatE. The RotatE model handles relationships... Consider it as a rotation in complex space, for a triple Its goal is to make ,in Complex vector embedding for head entity, relation, and tail entity. This represents the Hadamard product (element-by-element multiplication).

[0041] Its scoring function is defined as:

[0042] The model is trained by maximizing the scores of positive triples and minimizing the scores of negative triples. After training, we obtain a knowledge embedding library containing all entity and relation vectors, denoted as . .

[0043] Phase 2: Model Fine-tuning Based on Graph Vector Fusion Attention (200) The goal of this phase is to deeply integrate the structured knowledge obtained in Phase 1 into a large-scale language model to enhance its ability to understand traditional Chinese medicine literature. We propose a novel Graph-Vector Infused Attention (GVI-Attention) mechanism.

[0044] The core idea of ​​the GVI-Attention mechanism is to dynamically select from the self-attention calculation process of the standard Transformer. The knowledge vectors retrieved from [the database] are injected. Its detailed structure is as follows: Figure 2 As shown. For an input sequence of TCM text, before feeding it into the GVI-Attention layer, we first perform a lightweight entity linking step to identify entities in the text that are related to... The token or token span corresponding to the entity in the Chinese text. For example, for the sentence "This prescription uses Dioscorea opposita Thunb. as the monarch drug", "Dioscorea opposita Thunb." can be recognized as an entity. Suppose the input to a certain Transformer layer in a large language model is a series of token hidden layer representations , where . The processing flow of GVI-Attention is as follows: Knowledge vector retrieval (210): For the th token in the sequence, if it is recognized as part of an entity , the corresponding entity vector is retrieved from the knowledge embedding library . If the token does not belong to any entity, a learnable "null entity" vector is used.

[0045] Representation fusion (220): The original token hidden layer representation is fused with its corresponding entity vector to generate a knowledge-enhanced representation . The fusion process is achieved through a linear transformation with a non-linear activation to allow the model to learn how to best combine the two types of information: where and are learnable projection matrices used to map the original hidden layer representation and the entity vector to the same space. is the bias term. is used to stabilize the training process. This step is not simply concatenating vectors, but rather through a learnable weight matrix, allowing the model to autonomously determine the weights of the information from the text context and the structured information from the knowledge graph.

[0046] 3. Attention calculation (230): Use the fused representation sequence to calculate the standard self-attention. The query (Query), key (Key), and value (Value) matrices are generated by : where is the standard projection matrix for self-attention.

[0047] 2.2 Parameter-efficient fine-tuning To reduce the computational cost, we combine the GVI-Attention mechanism with parameter-efficient fine-tuning techniques (such as LoRA). Specifically, the main parameters of the large language model are kept frozen, and we only train the newly added projection matrices in GVI-Attention and the projection matrix applied to the standard attention projection matrixThe LoRA low-rank adaptation matrix. By fine-tuning on annotated datasets containing TCM literature with local characteristics, we finally obtained a powerful teacher model that, although the parameter increment is small, deeply integrates the TCM knowledge graph.

[0048] Phase 3: Knowledge-Guided Lightweight Distillation (300) While the teacher model boasts powerful performance, its large size makes it unsuitable for deployment on computationally limited primary healthcare equipment. The goal of this phase is to transfer its knowledge to a smaller student model while ensuring that critical knowledge reasoning capabilities are not lost. To this end, we designed a knowledge-guided distillation strategy. The core of this strategy is a composite loss function. It consists of three parts: 3.1 Standard distillation loss ( and ) Hard label loss ( This is the standard supervised learning loss, specifically the cross-entropy loss of the student model on the gold labels of the training data. It ensures that the student model learns basic task capabilities. in Y is the output probability of the student model, and Y_{} is the true label. (Soft label loss) This is the core of knowledge distillation: the student model learns to mimic the output probability distribution (soft labels) of the teacher model. This helps the student model learn the teacher model's "hidden knowledge" between categories. KL divergence is typically used to measure the difference between the two distributions.

[0049] in and The teacher model and the student model were respectively subjected to temperature coefficients. Smoothed output probability distribution.

[0050] 3.2 Loss of Spectral Structure Consistency ( (310) This is another core innovation of the distillation stage of this invention. Its purpose is to force the internal representation space of the student model to also reflect the structural relationships of the original knowledge graph, rather than just imitating the output of the teacher model. The specific implementation is as follows: For any sentence in the training data, if it contains a triple in the knowledge graph (For example, the sentence "Huai yam is the main treatment for spleen deficiency" corresponds to the three-element group (Huai yam, main treatment, spleen deficiency)), we perform the following operations: 1. Representation Extraction: Extract the head entity from the last or middle layer of the student model. Tail-end entity The corresponding word vector representation is denoted as and .

[0051] 2. Define relational transformation: We assume that in the representation space of the student model, the relation... It can also be modeled as a transformation operation. The simplest form is a linear translation transformation, that is, for each relation... Learning a relation vector Here These are parameters learned along with the student model during the distillation process.

[0052] 3. Calculate the loss: Ideally, Through relationships After the transformation, it should equal Therefore, we will Defined as the sum of the Euclidean distances (L2 norms) between the predicted tail entity representations and the true tail entity representations of all triples: in It is the set of all knowledge triples contained in the training data.

[0053] By minimizing this composite loss function The student model not only learns the input-output mapping of the teacher model, but is also "forced" to internally establish a representation space consistent with the structure of the TCM knowledge graph. This makes the lightweight student model more robust during reasoning and less prone to "illusions" that violate TCM theory.

[0054] System and Equipment Examples System embodiments of the present invention may include several functional modules configured on a server or cloud platform. For example... Figure 3 As shown, a typical electronic device (such as a server) 500 includes a processor 510, a memory 530, and a communication bus 540 for connecting them.

[0055] Memory 530: Used to store computer program instructions for executing the method of the present invention, as well as various types of data required during the execution of the method, such as a local traditional Chinese medicine literature corpus, a constructed hierarchical knowledge graph, and a knowledge vector embedding library. The trained teacher model and the final lightweight student model.

[0056] Processor 510: Responsible for executing program instructions in memory 530. When executing the program of this invention, the processor will sequentially call and run the corresponding functional modules: The knowledge graph construction module is responsible for performing the operations of Phase One, collecting data from original documents, and constructing the knowledge graph. And train to generate .

[0057] Model fine-tuning module: responsible for performing the operations of stage two, loading the pre-trained LLM, implementing the GVI-Attention mechanism, and fine-tuning on the labeled data to produce the teacher model.

[0058] Model distillation module: Responsible for executing the operations in Phase 3, loading the teacher model and student model architecture, implementing the knowledge-guided distillation process, training and producing the final lightweight student model.

[0059] Inference Service Module: Used to load lightweight student models and provide API services such as knowledge extraction, intelligent question answering, and assisted diagnosis.

[0060] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for fine-tuning and lightweight inference of a large-scale local TCM model integrating graph vector embedding, characterized in that, Includes the following steps: The steps for constructing a hierarchical TCM knowledge graph and generating graph vector embeddings are as follows: Based on the general TCM knowledge system and TCM literature from a specific region, a hierarchical knowledge graph containing a general knowledge layer and a local characteristic knowledge layer is constructed. The hierarchical knowledge graph is then trained using a knowledge graph representation learning algorithm to obtain the graph vector embeddings of each knowledge entity and knowledge relationship. The steps for fine-tuning a pre-trained large-scale language model through knowledge fusion are as follows: A graph vector fusion attention mechanism is used to fine-tune the pre-trained large-scale language model. The graph vectors are embedded and injected into the self-attention calculation module inside the model, so that the model integrates structured graph knowledge when processing input text, and a knowledge-enhanced teacher model is trained. The steps for knowledge-guided lightweight distillation are as follows: Design a composite loss function that includes graph structure consistency loss, and use knowledge distillation technology to transfer the knowledge of the teacher model to a student model with fewer parameters, thereby obtaining a lightweight model that retains the key knowledge reasoning ability.

2. The method for fine-tuning and lightweight inference of a large-scale local TCM model fused with graph vector embedding according to claim 1, characterized in that, The steps for constructing a hierarchical TCM knowledge graph include: constructing a general knowledge layer based on the national standard for TCM language system, containing core concepts of symptoms, prescriptions, herbs, and treatments and their interrelationships; mining and defining knowledge elements with regional characteristics from TCM literature in specific regions, including local entities and special relationships, as a local characteristic knowledge layer; and integrating the general knowledge layer and the local characteristic knowledge layer to form a unified hierarchical knowledge graph.

3. The method for fine-tuning and lightweight inference of a local TCM large-scale model with fused graph vector embedding as described in claim 1, characterized in that, The fine-tuning steps of the graph vector fusion attention mechanism specifically include: performing entity recognition on the input text to determine the lexical units or lexical units in the text that correspond to the knowledge entities in the hierarchical knowledge graph; for each lexical unit corresponding to the identified knowledge entity, fusing its hidden layer representation in the model with the corresponding entity vector retrieved from the graph vector embedding to generate a knowledge-enhanced lexical representation; using the knowledge-enhanced lexical representation to calculate the query, key, and value matrix in the self-attention mechanism, and completing the attention score calculation and weighted summation operation.

4. The method for fine-tuning and lightweight inference of a large-scale local TCM model fused with graph vector embedding according to claim 3, characterized in that, The knowledge-enhanced lexical representation The calculation formula is: in, For the model The original hidden layer representation of each word element, For the graph vector embedding of the knowledge entity corresponding to this word element, and For a learnable projection matrix, For bias terms, This is a layer normalization operation.

5. The method for fine-tuning and lightweight inference of a local TCM large-scale model with fused graph vector embedding as described in claim 1, characterized in that, In the step of performing knowledge-guided lightweight distillation, the composite loss function The expression is: in, The standard cross-entropy loss of the student model on the real labels; The KL divergence loss is used to compare the soft labels output by the student model and the teacher model. This represents the loss of consistency in the spectral structure. These are the weighting coefficients for each type of loss.

6. The method for fine-tuning and lightweight inference of a large-scale local TCM model fused with graph vector embedding according to claim 5, characterized in that, The loss of consistency of the map structure The calculation method is as follows: for each knowledge triple (head entity) that appears in the training data ,relation Tail entity Extract the vector representations of the head and tail entities from the hidden states of the student model. and ; Define a relation that simulates a student model representation space. Transformation operations This operation is designed to represent the head entity. Transformed into a predicted tail entity representation; the graph structure consistency loss Defined as the predicted tail entity representation and the true tail entity representation The distance between them is given by the formula: in, The set of all knowledge triples in the training set. This represents the square of the L2 norm.

7. A local TCM large-scale model fine-tuning and lightweight inference system integrating graph vector embedding, characterized in that, include: The knowledge graph construction module is used to construct a hierarchical knowledge graph containing a general knowledge layer and a local characteristic knowledge layer based on the general TCM knowledge system and TCM literature of a specific region. The hierarchical knowledge graph is trained using a knowledge graph representation learning algorithm to obtain the graph vector embedding of each knowledge entity and knowledge relationship. The model fine-tuning module is used to fine-tune the pre-trained large language model using a graph vector fusion attention mechanism. The graph vectors are embedded into the self-attention calculation module inside the model, so that the model integrates structured graph knowledge when processing input text, and trains a knowledge-enhanced teacher model. The model distillation module is used to design a composite loss function that includes graph structure consistency loss. It employs knowledge distillation technology to transfer the knowledge of the teacher model to the student model with fewer parameters, thereby obtaining a lightweight model that retains the ability to reason about knowledge.

8. The local TCM large-scale model fine-tuning and lightweight inference system with fusion graph vector embedding as described in claim 7, characterized in that, The model fine-tuning module is specifically used for: Entity recognition is performed on the input text to determine the lexical units or lexical units in the text that correspond to the knowledge entities in the hierarchical knowledge graph. For each lexical unit corresponding to the identified knowledge entity, its hidden layer representation in the model is fused with the corresponding entity vector retrieved from the graph vector embedding to generate a knowledge-enhanced lexical representation. The query, key, and value matrix in the self-attention mechanism is calculated using the knowledge-enhanced lexical representation, and attention score calculation and weighted summation operations are performed.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 6.

Images