Evidence perception long context refining method based on reinforcement learning

Abstract

The invention discloses an evidence perception long context refining method based on reinforcement learning. The method comprises the following steps: firstly, constructing a hierarchy-evidence matching matrix to obtain a state space, designing a discrete action space to operate an original medical record, and generating a refined context; step-by-step reasoning decision making is carried out according to a refining context and a coding decision big model, and a final diagnosis code is obtained; combining information extraction quality, format compliance and path accuracy to construct a composite reward and punishment function, and performing quantitative scoring on the refining behavior and the coding result; and finally, designing an asynchronous cooperative training mechanism based on double-layer optimization, finely adjusting the coding decision large model, and updating the refining module by adopting an actor-commentator framework. The method is suitable for a long medical record text ICD automatic coding task, and by designing a dynamic evidence perception and cooperative training mechanism, the precision of model long context processing and decision coding is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and medical artificial intelligence technology. Specifically, it relates to a text information extraction and long context processing technology, and more specifically, it relates to a long context refinement method based on reinforcement learning for evidence perception. Background Technology

[0002] As a crucial task in the fields of medical informatics and medical artificial intelligence, automated ICD coding has been recognized as a key component of hospital medical record management, medical insurance settlement, and epidemiological statistics. Its goal is to accurately extract patients' disease diagnosis information from unstructured, real-world clinical medical records and map it into standardized ICD codes. However, real-world clinical medical records often contain massive amounts of diagnostic and treatment information, generally characterized by length, uneven information density, and sparse distribution of key information. With increasingly refined requirements for ICD coding, the critical discriminative evidence supporting accurate decision-making is easily overwhelmed by massive amounts of clinical noise, and traditional information extraction and text classification methods face serious performance bottlenecks and generalization challenges.

[0003] Existing long text processing methods primarily rely on static truncation strategies or general text summarization techniques, aiming only to shorten the length of the input text. However, as the coding hierarchy deepens (e.g., refining from broad disease category chapters to subcategory levels with acute or chronic modifiers), the coding model's need for medical record evidence is constantly changing. Existing static truncation strategies or general text summarization methods struggle to match this dynamically changing evidence requirement, failing to achieve deep collaboration between the long context refinement process and the dynamic reasoning logic of the large coding decision model. They are also prone to losing discriminative evidence related to specific coding nodes, thus severely interfering with subsequent coding decision paths.

[0004] To address the challenges of dynamic processing of long texts, some recent studies have attempted to incorporate reinforcement learning techniques for the recombination and filtering of long texts. While this sequential decision-making approach improves the flexibility of text processing to some extent, it still faces numerous challenges in complex encoding decision-making scenarios. First, when reinforcement learning is used solely for long text refinement, the output is only intermediate text fragments. As an intermediate link in the overall encoding decision-making chain, the system struggles to independently and objectively evaluate the refinement quality of these fragments and their true contribution to the final encoding gain. This limitation of local evaluation easily leads to a deviation between local policy optimization and the global objective, resulting in an imbalance in evidence reliability and the risk of losing key semantics. Second, because the refinement module and the large encoding decision-making model are interdependent and dynamically interact during training, the model operates in a non-stationary environment. Existing conventional training methods struggle to overcome policy updates in non-stationary environments, causing model training to fail to converge or become trapped in local optima.

[0005] Based on the shortcomings of the existing technologies, this invention aims to solve the problems of traditional static processing methods being unable to adapt to the needs of dynamic evidence, the loss of key semantics due to the limitations of local evaluation in existing reinforcement learning methods, and the difficulty in model convergence caused by the interaction between the refinement module and the large model. In particular, it focuses on solving the problem of existing technical solutions being unable to adapt to the needs of dynamic evidence and the limitations of local evaluation, thereby achieving deep collaboration between long context refinement and dynamic encoding decision-making.

[0006] In conclusion, a new method for refining long contexts for evidence perception is needed. Summary of the Invention

[0007] In view of this, this invention provides an evidence-aware long-context refinement method based on reinforcement learning. First, a refinement policy modeling method based on evidence and hierarchy matching is designed. Its most significant feature is the construction of a hierarchy-evidence matching matrix, which integrates the semantics of the original medical record with the features of the current encoding hierarchy, ensuring the effectiveness of dynamic evidence extraction and medical record denoising. Second, an asynchronous collaborative training and composite reward / punishment method based on two-layer optimization is proposed. A result-oriented composite reward / punishment function combining process supervision and global reward is designed, and an actor-commentator framework is used to overcome the policy update challenge in non-stationary environments.

[0008] The technical approach of this invention is as follows: First, a hierarchical-evidence matching matrix is ​​constructed using the current encoding level features, historical evidence, and the semantics of the original medical records to obtain the state space representation of the refinement strategy module. Next, a discrete action space containing three types of operations—retention, mapping, and discarding—is designed. The original medical records are then manipulated through a policy network to generate a refinement context. Then, during the refinement and encoding decision-making process, a composite reward-penalty function is constructed using information extraction quality, format compliance, and path accuracy to quantitatively evaluate the refinement behavior. Finally, a two-layer optimized asynchronous collaborative training strategy is designed. By alternately fine-tuning the large encoding decision model and optimizing the policy gradient of the refinement module, the final dynamic long-context refinement strategy network is obtained. The specific steps are as follows:

[0009] S1. Input the real clinical medical record text into the refining module. The refining module performs operations on the fragments to obtain the refined contextual evidence fragments. The refining module includes: a state space, an action space, and a policy network. The state space integrates the original medical record semantics, the current encoding level features, and the historical evidence matching status. The action space covers three discrete operations: retention, term mapping, and loss. The policy network is used to select and execute the optimal discrete operation for the content of the state space.

[0010] S2. Based on the refined contextual evidence fragments, the encoding decision-making large model makes decisions level by level to obtain the final diagnostic code; the encoding decision-making large model is also known as the general large model, which makes choices at different encoding levels and makes top-down decisions along the ICD tree until the final diagnostic code is obtained; S3. Based on the asynchronous collaborative training mechanism of two-layer optimization, the parameters of the encoding decision-making large model are fine-tuned to provide a stable feedback environment for the refinement module; based on the contextual evidence fragments and the final diagnostic code generated in S1 and S2 as the exploration trajectory of reinforcement learning, the reward and punishment function is used for quantitative evaluation to generate a reward signal, and the policy gradient algorithm is used to iteratively update the policy network parameters until convergence is obtained to obtain the trained policy network.

[0011] S11. The state space of the refining module is specifically modeled as the state space S of the refining strategy module corresponding to the coding level t, formally defined as: ,in This represents the semantic feature vector of the segmented representation of the original medical record. The target vector is based on the feature dimension corresponding to the current ICD level t. The feature vectors that match historical evidence with the feature dimensions corresponding to the current level depend on the level-evidence matching matrix. , It records the cross-attention distribution between the historical evidence set and the feature dimensions corresponding to the current level, extracts the current highest score of each feature along the feature dimension direction, and finally constitutes... ;

[0012] S12, the action space of the refining module is designed as three types of discrete operations, formally defined as follows: Fragments are preserved For strongly related original medical record descriptions, direct extraction and retention are performed, along with term mapping. To standardize and retain abbreviations or colloquial expressions using a medical knowledge base, fragments are discarded. To filter out redundant noise segments that have no direct diagnostic value;

[0013] S13. The refined module strategy network selects the optimal discrete action for different medical record text fragments at different encoding levels t to obtain a refined and standard context.

[0014] S21. The aforementioned coding decision-making model, based on contextual evidence fragments, follows the ICD tree from the root node, sequentially passing through chapters, categories, and subcategories, and finally reaching the leaf node for coding, completing a top-down multi-level decision-making process.

[0015] S31. A result-oriented composite reward and punishment mechanism for information gain, which combines process supervision with global reward and proposes a result-oriented composite reward and punishment function for information gain.

[0016] S32, Information Extraction Quality Reward After each level of refinement, the quality of the segment after this level of refinement is formally defined as follows: ,in To adjust the weighting coefficients; As an indicator of evidence type completeness, it is a pre-defined set of target evidence for a specific coding level. Calculate the categories of evidence that are hit in the refined text. The proportion, in its formal definition, is ; As a core concept certainty indicator, based on the set of concept phrases P in the knowledge base, each refined piece of evidence is calculated. The semantic similarity with P is determined by a threshold. The formal definition of determining whether evidence contains sufficient information is... ;

[0017] S33, Format Compliance Rewards After each level of refinement, rule matching and evaluation are performed on the refined contextual evidence fragments based on the output specifications. This is formally defined as a piecewise function.

[0018]

[0019] in and These are the highest positive reward value and the lowest penalty reward value, respectively. is the number of missing fields, and l is the token length of the refined contextual evidence fragment. To preset the maximum token length, and These are the penalty coefficients for missing fields and the penalty coefficients for excessively long fields, respectively.

[0020] S34, Path Accuracy Reward : Utilize the topological structure of the ICD tree to calculate the encoding path obtained by the final encoding decision-making large model. The depth of the longest common prefix with the true path L is formally defined as: ;

[0021] S35. After each level of refinement and final decision, the system calculates the three sets of core rewards (S32, S33, and S34). The final composite reward and punishment function is defined as follows: ,in for Weight parameters within the range;

[0022] S36. Lock the aligned parameters of the large-scale coding decision model to serve as a stable feedback environment for the refinement module. An actor-critic framework is used to optimize the refinement module parameters. For each original medical record, the refinement module selects actions based on the current ICD level state and historical evidence sequence, generating refined text and injecting it into the input context of the large-scale coding decision model. This process is repeated until the large-scale coding decision model completes the full path exploration from the root node to the leaf node on the ICD coding tree, recording the complete action trajectory. This serves as the actor's strategy. The critic uses the composite reward and punishment function in S35 to evaluate and score the entire refinement process. Based on the critic's score, the actor updates the refinement module in reverse through the returned global reward signal, dynamically adjusting the action strategy, ultimately obtaining the refinement strategy network.

[0023] S111, the aforementioned feature dimension target vector It possesses hierarchical awareness and has different semantic projection directions at different coding levels. At shallow levels, the projection direction focuses on the overall disease category, while at deep levels, it focuses on local fine-grained information to capture information categories with pathological attributes.

[0024] S112, the aforementioned hierarchical-evidence matching matrix The construction includes: calculating the cross-attention distribution between the historical extracted evidence set and the feature dimension of the current decision level, and extracting the highest score along the feature dimension direction to identify the optimal evidence required for the current level.

[0025] This invention designs an evidence-aware long-context refinement framework based on reinforcement learning. By constructing a hierarchical evidence matching matrix and a discrete action space, it breaks through the limitations of traditional static truncation or general summarization methods. It can dynamically capture discriminative evidence and effectively filter redundant clinical noise as the ICD coding decision level deepens. Simultaneously, this invention designs a result-oriented composite reward and punishment mechanism that integrates information extraction quality, format compliance, and final path accuracy. By combining process supervision with global constraints, it effectively overcomes the limitations of local evaluation and the risk of losing key semantics in long text refinement, avoiding deviation between local strategy optimization and the global objective. Furthermore, addressing the non-stationary environment caused by the dynamic interaction between the refinement module and the large coding decision model, this invention designs an asynchronous collaborative training strategy based on two-layer optimization. It utilizes alternating fine-tuning of the decision benchmark environment and the actor-commentator framework to optimize the refinement strategy, successfully solving the challenges of strategy updating and model convergence in non-stationary environments. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall framework of the evidence-aware long context refinement method based on reinforcement learning in this invention;

[0027] Figure 2This is a flowchart of the evidence-aware long context refinement mechanism based on reinforcement learning in this invention; Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention and to make the above-mentioned objectives, technical solutions and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to embodiments.

[0029] Please see Figure 1 , Figure 1 This is a schematic diagram of the overall framework of the evidence-aware long context refinement method based on reinforcement learning according to the present invention. The method includes: first, inputting real clinical medical record text into the refinement module, which performs operations on the fragments to obtain contextual evidence fragments; second, based on the contextual evidence fragments, the encoding decision-making big model completes the current level decision encoding, and so on, until the final diagnosis encoding is obtained. The above is an overview of the method's implementation and usage. However, because the policy network of the refinement module has not been trained, resulting in poor performance, this invention models the above forward execution process as an exploration process of reinforcement learning, using a composite reward-penalty function as the global feedback signal for reinforcement learning. Through a two-layer optimized asynchronous collaborative training mechanism, the encoding decision-making big model and the policy gradient algorithm are fine-tuned to optimize the policy network, ultimately obtaining the trained policy network.

[0030] Please see Figure 2 , Figure 2 This is a flowchart of the evidence-aware long context refinement mechanism based on reinforcement learning according to the present invention. The specific steps include:

[0031] S1. Input the real clinical medical record text into the refining module. The refining module performs operations on the fragments to obtain the refined contextual evidence fragments. The refining module includes: a state space, an action space, and a policy network. The state space integrates the original medical record semantics, the current encoding level features, and the historical evidence matching status. The action space covers three discrete operations: retention, term mapping, and loss. The policy network is used to select and execute the optimal discrete operation for the content of the state space.

[0032] S2. Based on the refined contextual evidence fragments, the encoding decision-making model makes decisions step by step to obtain the final diagnostic code. The encoding decision-making model is the general model, which in specific implementation can be qwen30b-a3b. It makes choices at different encoding levels, making top-down decisions along the ICD tree from chapter, category, subcategory, and leaf node encoding until the final diagnostic code is obtained. S3. Based on a two-layer optimized asynchronous collaborative training mechanism, the parameters of the encoding decision-making model are fine-tuned to provide a stable feedback environment for the refinement module. Using the contextual evidence fragments and the final diagnostic code generated in S1 and S2 as the exploration trajectory for reinforcement learning, a reward / penalty function is used for quantitative evaluation to generate a reward signal. The policy gradient algorithm is used to iteratively update the policy network parameters until convergence is achieved, resulting in a trained policy network.

[0033] S11. The state space of the refining module is specifically modeled as the state space S of the refining strategy module corresponding to the coding level t, formally defined as: ,in This represents the semantic feature vector of the segmented representation of the original medical record. The target vector is determined based on the feature dimension corresponding to the current ICD level t: for example, it is projected to a broad semantic dimension of major diseases (respiratory system, cardiovascular system, etc.) in the chapter-level feature space, and to more detailed modifier features (acute, chronic, etc.) in the subcategory-level feature space. The feature vectors that match historical evidence with the feature dimensions corresponding to the current level depend on the level-evidence matching matrix. , It records the cross-attention distribution between the historical evidence set and the feature dimensions corresponding to the current level, extracts the current highest score of each feature along the feature dimension direction, and finally constitutes... ;

[0034] S12, the action space of the refining module is designed as three types of discrete operations, formally defined as follows: Fragments are preserved For strongly related original medical record descriptions, direct extraction and retention are performed, along with term mapping. To standardize and retain abbreviations or colloquial expressions using a medical knowledge base, fragments are discarded. To filter out redundant noise segments that have no direct diagnostic value;

[0035] S13. The refined module strategy network selects the optimal discrete action for different medical record text fragments at different encoding levels t to obtain a refined and standard context.

[0036] S21. The aforementioned coding decision-making model, based on contextual evidence fragments, follows the ICD tree from the root node, sequentially passing through chapters, categories, and subcategories, and finally reaching the leaf node for coding, completing a top-down multi-level decision-making process.

[0037] S31. A result-oriented composite reward and punishment mechanism for information gain, which combines process supervision with global reward and proposes a result-oriented composite reward and punishment function for information gain.

[0038] S32, Information Extraction Quality Reward After each level of refinement, the quality of the segment after this level of refinement is formally defined as follows: ,in To adjust the weighting coefficients; As an indicator of evidence type completeness, it is a pre-defined set of target evidence for a specific coding level. Calculate the categories of evidence that are hit in the refined text. The proportion, in its formal definition, is ; As a core concept certainty indicator, based on the set of concept phrases P in the knowledge base, each refined piece of evidence is calculated. The semantic similarity with P is determined by a threshold. The formal definition of determining whether evidence contains sufficient information is... ;

[0039] S33, Format Compliance Rewards After each level of refinement, rule matching and evaluation are performed on the refined contextual evidence fragments based on the output specifications. This is formally defined as a piecewise function.

[0040]

[0041] in and These are the highest positive reward value and the lowest penalty reward value, respectively. is the number of missing fields, and l is the token length of the refined contextual evidence fragment. To preset the maximum token length, and These are the penalty coefficients for missing fields and the penalty coefficients for excessively long fields, respectively.

[0042] S34, Path Accuracy Reward : Utilize the topological structure of the ICD tree to calculate the encoding path obtained by the final encoding decision-making large model. The depth of the longest common prefix with the true path L is formally defined as: ;

[0043] S35. After each level of refinement and final decision, the system calculates the three sets of core rewards (S32, S33, and S34). The final composite reward and punishment function is defined as follows: ,in for Weight parameters within the range;

[0044] S36. Lock the aligned parameters of the large-scale coding decision model to serve as a stable feedback environment for the refinement module. An actor-critic framework is used to optimize the refinement module parameters. For each original medical record, the refinement module selects actions based on the current ICD level state and historical evidence sequence, generating refined text and injecting it into the input context of the large-scale coding decision model. This process is repeated until the large-scale coding decision model completes the full path exploration from the root node to the leaf node on the ICD coding tree, recording the complete action trajectory. This serves as the actor's strategy. The critic uses the composite reward and punishment function in S35 to evaluate and score the entire refinement process. Based on the critic's score, the actor updates the refinement module in reverse through the returned global reward signal, dynamically adjusting the action strategy, ultimately obtaining the refinement strategy network.

[0045] S111, the aforementioned feature dimension target vector It possesses hierarchical awareness and has different semantic projection directions at different coding levels. At shallow levels, the projection direction focuses on the overall disease category, while at deep levels, it focuses on local fine-grained information to capture information categories with pathological attributes.

[0046] S112, the aforementioned hierarchical-evidence matching matrix The construction includes: calculating the cross-attention distribution between the historical extracted evidence set and the feature dimension of the current decision level, and extracting the highest score along the feature dimension direction to identify the optimal evidence required for the current level.

[0047] Specific embodiments of the present invention have been described above with reference to the accompanying drawings. However, those skilled in the art will understand that various modifications and substitutions can be made to the specific embodiments of the present invention without departing from the spirit and scope of the invention. All such modifications and substitutions fall within the scope defined by the claims of the present invention.

Claims

1. A method for evidence-aware long context refinement based on reinforcement learning, characterized in that... Includes the following steps: S1. Input the real clinical medical record text into the refining module. The refining module performs operations on the fragments to obtain the refined contextual evidence fragments. The refining module includes: a state space, an action space, and a policy network. The state space integrates the original medical record semantics, the current encoding level features, and the historical evidence matching status. The action space covers three discrete operations: retention, term mapping, and loss. The policy network is used to select and execute the optimal discrete operation for the content of the state space. S2. Based on the refined contextual evidence fragments, encode the decision-making model step-by-step decision codes to obtain the final diagnostic code; The large-scale encoding decision model, also known as the general large-scale model, makes choices at different encoding levels, making top-down decisions along the ICD tree until the final diagnostic code is obtained. S3, an asynchronous collaborative training mechanism based on two-layer optimization, fine-tunes the parameters of the large-scale encoding decision model, providing a stable feedback environment for the refining module. Based on the contextual evidence fragments generated by S1 and S2 and the final diagnostic code as the exploration trajectory of reinforcement learning, a reward and punishment function is used for quantitative evaluation to generate a reward signal. The policy gradient algorithm is used to iteratively update the policy network parameters until convergence is obtained to obtain the trained policy network.

2. The evidence-aware long context refinement method based on reinforcement learning according to claim 1, characterized in that, The specific steps of step S1 include: S11. The state space of the refining module is specifically modeled as the state space S of the refining strategy module corresponding to the coding level t, formally defined as: ,in This represents the semantic feature vector of the segmented representation of the original medical record. The target vector is based on the feature dimension corresponding to the current ICD level t. The feature vectors that match historical evidence with the feature dimensions corresponding to the current level depend on the level-evidence matching matrix. , It records the cross-attention distribution between the historical evidence set and the feature dimensions corresponding to the current level, extracts the current highest score of each feature along the feature dimension direction, and finally constitutes ; S12, the action space of the refining module is designed as three types of discrete operations, formally defined as follows: Fragments are preserved For strongly related original medical record descriptions, direct extraction and retention are performed, along with term mapping. To standardize and retain abbreviations or colloquial expressions using a medical knowledge base, fragments are discarded. To filter out redundant noise segments that have no direct diagnostic value; S13. The refined module strategy network selects the optimal discrete action for different medical record text fragments at different encoding levels t to obtain a refined and standard context.

3. The evidence-aware long context refinement method based on reinforcement learning according to claim 1, characterized in that, The specific steps of step S2 include: S21. The aforementioned coding decision-making model, based on contextual evidence fragments, follows the ICD tree from the root node, sequentially passing through chapters, categories, and subcategories, and finally reaching the leaf node for coding, completing a top-down multi-level decision-making process.

4. The evidence-aware long context refinement method based on reinforcement learning according to claim 1, characterized in that, The specific steps of step S3 include: S31. A result-oriented composite reward and punishment mechanism for information gain, which combines process supervision with global reward and proposes a result-oriented composite reward and punishment function for information gain. S32, Information Extraction Quality Reward After each level of refinement, the quality of the segment after this level of refinement is formally defined as follows: ,in The adjustment weight coefficient is within the range of (0,1); As an indicator of evidence type completeness, it is a pre-defined set of target evidence for a specific coding level. Calculate the categories of evidence that are hit in the refined text. The proportion, in its formal definition, is ; As a core concept certainty indicator, based on the set of concept phrases P in the knowledge base, each refined piece of evidence is calculated. The semantic similarity with the set of concept phrases P is determined by a threshold. The formal definition of determining whether evidence contains sufficient information is... ; S33, Format Compliance Rewards After each level of refinement, rule matching and evaluation are performed on the refined contextual evidence fragments based on the output specifications. This is formally defined as a piecewise function. in and These are the highest positive reward value and the lowest penalty reward value, respectively. is the number of missing fields, and l is the token length of the refined contextual evidence fragment. To preset the maximum token length, and These are the penalty coefficients for missing fields and the penalty coefficients for excessively long fields, respectively. S34, Path Accuracy Reward : Utilize the topological structure of the ICD tree to calculate the encoding path obtained by the final encoding decision-making large model. The depth of the longest common prefix with the true path L is formally defined as: ,in The scaling factor is within the range of [1,3]. S35. After each level of refinement and final decision, the system calculates the three sets of core rewards (S32, S33, and S34). The final composite reward and punishment function is defined as follows: ,in for Weight parameters within the range; S36. Lock the aligned parameters of the large-scale coding decision model to serve as a stable feedback environment for the refinement module. An actor-critic framework is used to optimize the refinement module parameters. For each original medical record, the refinement module selects actions based on the current ICD level state and historical evidence sequence, generating refined text and injecting it into the input context of the large-scale coding decision model. This process is repeated until the large-scale coding decision model completes the full path exploration from the root node to the leaf node on the ICD coding tree, recording the complete action trajectory. This serves as the actor's strategy. The critic uses the composite reward and punishment function in S35 to evaluate and score the entire refinement process. Based on the critic's score, the actor updates the refinement module in reverse through the returned global reward signal, dynamically adjusting the action strategy, ultimately obtaining the refinement strategy network.

5. The specific steps according to claim 2, characterized in that... S11 specifically includes: S111, the aforementioned feature dimension target vector It possesses hierarchical awareness and has different semantic projection directions at different coding levels. At shallow levels, the projection direction focuses on the overall disease category, while at deep levels, it focuses on local fine-grained information to capture information categories with pathological attributes. S112, the aforementioned hierarchical-evidence matching matrix The construction includes: calculating the cross-attention distribution between the historical extracted evidence set and the feature dimension of the current decision level, and extracting the highest score along the feature dimension direction to identify the optimal evidence required for the current level.

Images