Medical decision enhancement method based on gated recursive large model architecture
By introducing recursive attention layer and knowledge graph training into the large language model, the problems of fuzzy and missing information in medical decision-making are solved, more in-depth information-related learning and decision-making support are achieved, and the accuracy and efficiency of medical decision-making are improved.
Patent Information
- Application Number
- CN202510618502.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-14
- Publication Date
- 2025-08-15
AI Technical Summary
Existing large language models are difficult to deal with decision-making difficulties caused by fuzzy and missing information in medical decision-making, and traditional architectures are difficult to effectively learn and distinguish complex medical information.
The gated recursive big model architecture is adopted, and the attention layer of the big model is replaced with the recursive attention layer, and the knowledge graph and patient information data set are trained. The input information is processed through the recursive and self-attention modules, and the recursive depth is controlled by global entropy to achieve deep correlation learning.
It improves the analytical capabilities of large language models in medical decision-making, realizes end-to-end medical decision-making support, and improves the accuracy and efficiency of decision-making.
Smart Images
Figure CN120496872A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of medical decision technology and relates to a medical decision enhancement method based on a gated recursive large model architecture. Background Art
[0002] The world is currently facing a global shortage of medical personnel. Advances in artificial intelligence (AI) technology have brought new, low-cost methods to assist medical decision-making. To improve the effectiveness of medical decision-making, Shang et al. constructed a graph convolutional neural network (GAMENet). This model uses a dynamic graph-enhanced memory module to capture longitudinal patient admission records and, combined with drug maps, optimizes the safety of medication decisions. While graph convolutional neural network-based methods effectively handle inter-drug correlations, they often overlook the semantic information in medical data and rely excessively on manual extraction of structured data, which remains labor-intensive. Therefore, Liu et al. proposed the LEADER model, which combines drug maps with a large language model to inject contextual semantic information, improving the model's global semantic understanding and enhancing medical decision-making. Gao et al. combined knowledge graphs with graph retrieval enhancement and a large language model to enhance the model's capabilities in automated diagnosis. While these methods leverage the powerful reasoning capabilities of large language models to improve decision accuracy, they still face bottlenecks. Based on traditional large language model architectures, they struggle to handle decision-making difficulties caused by ambiguous or missing information. This is primarily due to the large language model's inability to learn difficult-to-distinguish decisions. Therefore, to overcome this problem, it is necessary to design a new large language model network architecture to strengthen the model's learning of this part. Summary of the Invention
[0003] In view of this, the object of the present invention is to provide a medical decision enhancement method based on a gated recursive large model architecture.
[0004] In order to achieve the above object, the present invention provides the following technical solutions:
[0005] A medical decision enhancement method based on a gated recursive large model architecture includes the following steps:
[0006] Replace the attention layer in the large model with a recurrent attention layer;
[0007] Use knowledge graphs and patient information datasets to train and optimize the improved large model;
[0008] Deploy the trained and parameter-tuned large model, specify the knowledge graph to be loaded, and then input patient information to obtain the best medical decision.
[0009] Furthermore, after replacing the large model with the recursive attention layer, the data processing steps for patient information are as follows:
[0010] Step 1: Perform word embedding and entity extraction on the original input x, perform graph retrieval and graph embedding on the extracted entities, and then perform matrix addition and normalization on the word embedding and graph embedding to obtain the new input X';
[0011] Step 2: Set the number of recursions t = 1 and input X' into the self-attention module for processing;
[0012] Step 3: Normalize the output of the self-attention module and then process it using the feedforward network. The processed result and the output of the self-attention module are matrix-added to obtain the gate input matrix Z, which then enters the gate module.
[0013] Step 4: The gating module compares the global entropy e of the gated input matrix Z with the threshold τ; if e < τ, it proceeds to step S5; if e > τ, it performs the following recursive operations:
[0014] The number of recursions t is counted plus 1;
[0015] Pass the gated input matrix Z back to the normalized input part and perform matrix addition with the normalized input X' to obtain the new input
[0016] X t 'Input the self-attention module and the recursive attention module for collaborative processing;
[0017] Repeat the above recursive operation until e<τ, or t=t max , t max When the maximum number of recursive rounds is reached, the recursion is stopped and the process goes to step S5;
[0018] Step 5: Use the output as the input of the next recursive attention layer until the last recursive attention layer is processed; each recursive attention layer contains the contents of steps 2-4;
[0019] Step 6: Normalize the output of the last recursive attention layer and then process it through a linear layer and softmax function to obtain the final output result.
[0020] Furthermore, step 1 specifically includes the following steps:
[0021] Step 1.1: Perform entity extraction on the original input x and perform graph retrieval based on the extracted entities to obtain a matching subgraph g = Φ(x), where Φ represents the graph retrieval method;
[0022] Step 1.2: Perform word embedding on the original input x to obtain X;
[0023] Step 1.3: Embed the matching subgraph g to obtain G;
[0024] Step 1.4: Keep the vector mapping dimensions of word embedding and graph embedding consistent for matrix addition;
[0025] Step 1.5: Perform matrix addition and normalize to get the new input Where w is the word length and d is the vector mapping dimension.
[0026] Furthermore, the specific calculation formulas for the query vector Q, key vector K, and value vector V of the self-attention module are as follows:
[0027]
[0028] V=X'W V
[0029] Where Q0, K0 are Q, K at t = 1; W Q ,W K ,W V are the weight matrices corresponding to Q, K, and V, respectively. and are the rotation position encodings of Q and K, respectively, m and n are the row vector positions corresponding to the input word embedding X', and Θ is the encoding spacing.
[0030] Furthermore, the global entropy e in step 4 is calculated as follows:
[0031] Integrate the gated input matrix Z through a linear layer:
[0032] Y FNN =W FNN Z
[0033] where Y FNN is the integrated matrix, Where b is the total number of words in the dictionary,
[0034] Calculate the probability distribution vector P of the gate input matrix Z:
[0035] P=softmax(Y FNN )
[0036] in
[0037] Calculate the global entropy e:
[0038]
[0039] where the subscript i represents the i-th element of P.
[0040] Furthermore, the self-attention module and the recursive attention module in step S4 work together, where the specific calculation steps of Q, K, V and attention a are:
[0041] Step 4.1: X t 'As input, calculate the recursive self-attention:
[0042]
[0043] V=V
[0044]
[0045] V remains the same as in the first round, and the weight matrix also remains the same as in the first round, only the input is changed;
[0046] Step 4.2: Calculate recursive attention:
[0047]
[0048] Step 4.3: Calculate the raw self-attention:
[0049]
[0050] Step 4.4: Calculate the attention a under the current recursive round number:
[0051]
[0052] Furthermore, the improved large model is trained and optimized using the knowledge graph and patient information dataset, specifically including:
[0053] Construct a patient information dataset and divide it into a training set and a validation set. The labels of the training set and validation set are the medical decisions actually adopted by medical institutions.
[0054] Constructing a knowledge graph, wherein the knowledge graph includes drug knowledge and medical knowledge;
[0055] Use the knowledge graph and training set to train the improved large model;
[0056] The maximum number of recursive rounds t on the validation set max And the hyperparameter gating threshold τ is searched and optimized to obtain the optimal parameter value.
[0057] The beneficial effects of the present invention are as follows: In response to the problem that current methods are not effective in distinguishing medical decisions, the present invention proposes a method based on a gated recursive large language model architecture, which quantifies the difficulty of the decision by generating the entropy of the decision through the large model, and then controls whether to use recursive deep learning through the gating module. The uncertain decision information is recursively mapped to the deep space, and the deep correlation in the decision information is learned by combining recursive attention and self-attention to improve the medical decision-making ability of the large language model. This technology provides a new architecture for medical decision-making tasks, enhances the in-depth analysis ability of the large language model for decision information, and improves the application level of medical decision-making based on the large language model. End-to-end medical decision support is achieved.
[0058] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:
[0060] Figure 1 This is the overall structure diagram of the gated recursive attention model. DETAILED DESCRIPTION
[0061] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.
[0062] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.
[0063] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.
[0064] Example 1:
[0065] The present invention provides a medical decision enhancement method based on a gated recursive large model architecture, comprising the following steps:
[0066] Step 1: Construct a patient information dataset. This example uses the MIMIC-III dataset, which includes various patient medical records, to construct a knowledge graph. In this example, the knowledge graph uses DiseaseKG, which contains drug knowledge and medical knowledge and is stored in JSON format. All patient information datasets are divided into training and validation sets. The labels of the training and validation sets are the medical decisions actually taken by the medical institutions, such as referrals, discharges, medications, and examinations. The knowledge graph is not divided into sets.
[0067] Step 2: Select a large model base. This example uses LLama3-8B-Chat as the base model.
[0068] Step 3: Replace the attention layer of the selected base model with a recursive attention layer;
[0069] Step 4: Train the LLama3-8B-Chat model using the knowledge graph and patient information data from the training set;
[0070] Furthermore, in order to achieve the best medical decision-making effect, the maximum number of recursive rounds t is set on the validation set. max And the hyperparameter gating threshold τ is searched and optimized. In this embodiment, grid search optimization is used to obtain the optimal parameter value, t max The search range of is 8-32 with an interval of 1, and the search range of τ is 0.1-0.9 with an interval of 0.1;
[0071] Finally, the trained and tuned model weights are saved locally. In this example, they are saved in the .safetensor format to facilitate loading and reasoning with large-scale model inference programs such as huggingface and modelscope. When using them, the weights are loaded through huggingface or modelscope, the knowledge graph to be loaded is specified, and then the patient information is entered to obtain the best medical decision.
[0072] Example 2:
[0073] like Figure 1 As shown, this embodiment provides a detailed structure of the gated recursive attention model, and its working principle is as follows:
[0074] First, perform entity extraction on the original input x and perform graph retrieval based on the extracted entities to obtain a matching subgraph g = Φ(x), where Φ represents the graph retrieval method. A word embedding is performed on the original input to obtain X, and a graph embedding is performed on the matching subgraph to obtain G. The two embeddings keep the vector mapping dimensions consistent for matrix addition. After performing matrix addition and normalization, the new input is obtained. Where w is the word length, d is the vector mapping dimension. t is used to represent the number of recursive rounds, t max The maximum number of recursive rounds set for the hyperparameter is t = 1. When t = 1, only the self-attention module works. The specific calculation formulas for the query vector Q, key vector K, and value vector V are:
[0075]
[0076] V=X'W V
[0077] Where Q0, K0 are Q and K at t = 1. Q ,W K ,W V are the weight matrices corresponding to Q, K, and V, respectively. and where m and n are the rotational position encodings of Q and K, respectively. m and n are the row vector positions corresponding to the input word embedding X', respectively. Θ is the encoding spacing. The rest of the process is the same as the traditional attention module. The feedforward network is then used for processing. The processed result and the output of the self-attention module are matrix-added to obtain the gated input matrix Z. The gating module is connected after the feedforward network. The specific workflow of this gating module is as follows:
[0078] Integrate the gated input matrix Z through a linear layer:
[0079] Y FNN =W FNN Z
[0080] where Y FNN is the integrated matrix, Where b is the total number of words in the dictionary,
[0081] Calculate the probability distribution vector P of the gate input matrix Z:
[0082] P=softmax(Y FNN )
[0083] in
[0084] Calculate the global entropy e:
[0085]
[0086] where the subscript i represents the i-th element of P.
[0087] Make a threshold judgment:
[0088] Set the hyperparameter τ. If e<τ, output directly. If e>τ, perform a recursive operation, pass the gated input matrix Z back to the normalized input part, and perform matrix addition with the normalized input X' to obtain a new input. When the gating module is activated, that is, e>τ, the t count is increased by 1, and the next round of recursion is entered. At this time, the self-attention module and the recursive attention module work together. The specific calculation steps of Q, K, V and attention a are:
[0089] 1. X t 'As input, calculate the recursive self-attention:
[0090]
[0091] V remains the same as in the first round, and the weight matrix also remains the same as in the first round, only the input is changed.
[0092] 2. Calculate recursive attention:
[0093]
[0094] 3. Calculate the original self-attention:
[0095]
[0096] 4. Calculate the attention a under the current recursive round number:
[0097]
[0098] The subsequent steps are the same as the first round, and the weight parameter updates are inherited under the new round number. When e<τ is satisfied, the recursion stops and the output is sent to the next recursive attention layer. Figure 1 The N× in the equation represents a total of N recursive attention layers connected in series. Note that a maximum of t max Gated recursion prevents memory overflow caused by excessive thinking, ensuring operational efficiency within an appropriate range. Through gated recursion, large models can conduct in-depth learning on uncertain decisions to improve decision accuracy.
[0099] In the above embodiments, references to "this embodiment" in the specification indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily refer to the same embodiment.
[0100] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory structures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed. The embodiments of the present invention are intended to encompass all such alternatives, modifications, and variations that fall within the broad scope of the appended claims.
[0101] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in this embodiment is implemented.
[0102] This embodiment also provides an electronic terminal, including: a processor and a memory;
[0103] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods in this embodiment.
[0104] Regarding the computer-readable storage medium in this embodiment, those skilled in the art will appreciate that all or part of the steps in the aforementioned method embodiments can be implemented using hardware associated with the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps in the aforementioned method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0105] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used for communication, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes the various steps of the above method.
[0106] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.
[0107] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.
[0108] The present invention can be used in a wide variety of general-purpose or special-purpose computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above.
[0109] The present invention may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.
[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.
Claims
1. A medical decision enhancement method based on a gated recursive large model architecture, characterized by: The following steps are involved: Replace the attention layer in the large model with a recurrent attention layer; Use knowledge graphs and patient information datasets to train and optimize the improved large model; Deploy the trained and parameter-tuned large model, specify the knowledge graph to be loaded, and then input patient information to obtain the best medical decision.
2. The medical decision enhancement method based on the gated recursive large model architecture according to claim 1 is characterized by: After replacing the large model with the recursive attention layer, the data processing steps for patient information are as follows: Step 1: Perform word embedding and entity extraction on the original input x, perform graph retrieval and graph embedding on the extracted entities, and then perform matrix addition and normalization on the word embedding and graph embedding to obtain the new input X'; Step 2: Set the number of recursions t = 1 and input X' into the self-attention module for processing; Step 3: Normalize the output of the self-attention module and then process it using the feedforward network. The processed result and the output of the self-attention module are matrix-added to obtain the gate input matrix Z, which then enters the gate module. Step 4: The gating module compares the global entropy e of the gated input matrix Z with the threshold τ; if e < τ, it proceeds to step S5; if e > τ, it performs the following recursive operations: The number of recursions t is counted plus 1; Pass the gated input matrix Z back to the normalized input part and perform matrix addition with the normalized input X′ to obtain the new input X t 'Input the self-attention module and the recursive attention module for collaborative processing; Repeat the above recursive operation until e<τ, or t=t max , t max When the maximum number of recursive rounds is reached, the recursion is stopped and the process goes to step S5; Step 5: Use the output result as the input of the next recursive attention layer until the last recursive attention layer is processed; Each recursive attention layer contains the contents of steps 2-4; Step 6: Normalize the output of the last recursive attention layer and then process it through a linear layer and softmax function to obtain the final output result.
3. The medical decision enhancement method based on the gated recursive large model architecture according to claim 2 is characterized by: Step 1 specifically includes the following steps: Step 1.1: Perform entity extraction on the original input x and perform graph retrieval based on the extracted entities to obtain a matching subgraph g = Φ(x), where Φ represents the graph retrieval method; Step 1.2: Perform word embedding on the original input x to obtain X; Step 1.3: Embed the matching subgraph g to obtain G; Step 1.4: Keep the vector mapping dimensions of word embedding and graph embedding consistent for matrix addition; Step 1.5: Perform matrix addition and normalize to get the new input Where w is the word length and d is the vector mapping dimension.
4. The medical decision enhancement method based on a gated recursive large model architecture according to claim 2, characterized in that: The specific calculation formula of the query vector Q, key vector K, and value vector V of the self-attention module is: V=X′W V Where Q0, K0 are Q, K at t = 1; W Q ,W K ,W V are the weight matrices corresponding to Q, K, and V, respectively. and are the rotation position encodings of Q and K, respectively, m and n are the row vector positions corresponding to the input word embedding X′, and Θ is the encoding spacing.
5. The medical decision enhancement method based on a gated recursive large model architecture according to claim 2, characterized in that: The global entropy e in step 4 is calculated as follows: Integrate the gated input matrix Z through a linear layer: Y FNN =W FNN Z where Y FNN is the integrated matrix, Where b is the total number of words in the dictionary, Calculate the probability distribution vector P of the gate input matrix Z: P=softmax(Y FNN ) in Calculate the global entropy e: where the subscript i represents the i-th element of P.
6. The medical decision enhancement method based on a gated recursive large model architecture according to claim 5 is characterized by: The self-attention module and the recursive attention module in step S4 work together, where the specific calculation steps of Q, K, V and attention a are: Step 4.1: X t 'As input, calculate the recursive self-attention: V remains the same as in the first round, and the weight matrix also remains the same as in the first round, only the input is changed; Step 4.2: Calculate recursive attention: Step 4.3: Calculate the raw self-attention: Step 4.4: Calculate the attention a under the current recursive round number:
7. The medical decision enhancement method based on a gated recursive large model architecture according to claim 1 is characterized by: The improved large model is trained and optimized using the knowledge graph and patient information dataset, specifically including: Construct a patient information dataset and divide it into a training set and a validation set. The labels of the training set and validation set are the medical decisions actually adopted by medical institutions. Constructing a knowledge graph, wherein the knowledge graph includes drug knowledge and medical knowledge; Use the knowledge graph and training set to train the improved large model; The maximum number of recursive rounds t on the validation set max And the hyperparameter gating threshold τ is searched and optimized to obtain the optimal parameter value.