Intensive care parameter dynamic adjustment method based on reinforcement learning

Abstract

The present application relates to the technical field of computer information processing, in particular to a dynamic adjustment method of intensive care parameters based on reinforcement learning. The scheme comprises: obtaining patient respiratory mechanics parameter data and continuous invasive arterial blood pressure waveform data; generating a mechanics context vector representing the real-time influence mode and intensity of respiratory mechanics on blood pressure through a mechanics context encoder based on a dual-channel neural network with attention mechanism; inputting the vector and the patient physiological state feature vector into a reinforcement learning agent after concatenation, and outputting an adjustment action of vasoactive drug infusion rate to control the infusion pump to execute; the reward function of the reinforcement learning agent adopts a dynamic target blood pressure value as a reference which decreases with the increase of positive end-expiratory pressure. The method is mainly used for closed-loop automatic adjustment of vasoactive drugs for mechanically ventilated patients in intensive care, to suppress non-pharmacological interference and avoid excessive drug administration.

Description

Technical Field

[0001] This invention relates to the field of computer information processing technology, and more specifically, to a method for dynamic adjustment of intensive care parameters based on reinforcement learning. Background Technology

[0002] Reinforcement learning, an important branch of machine learning, enables agents to autonomously discover optimal control strategies in complex dynamic systems through continuous interaction and trial-and-error learning with their environment. In a typical reinforcement learning framework, the agent obtains the current state from the environment, outputs control actions based on its internal policy, and the environment then provides feedback on the next state and reward signal. The agent optimizes its policy accordingly to maximize the cumulative reward.

[0003] However, existing reinforcement learning control systems exhibit significant performance degradation when faced with a common type of physically coupled scenario. Specifically, when an external physical operation produces a predictable coupling effect on the controlled state of the system in accordance with physical laws, and this effect is numerically indistinguishable from the changes caused by the deterioration of the system's internal characteristics, existing reinforcement learning systems will output incorrect compensatory control actions due to the loss of the ability to identify the root cause of the state change, thereby causing deviation from the control objective or even system oscillation.

[0004] This signal ambiguity problem caused by external physical coupling is particularly prominent in the field of continuous process control. Taking a complex fluid transport system as an example, when an operator adjusts the opening of upstream pipeline valves or the pump pressure to achieve an upstream process objective (such as maintaining the container's resistance to negative pressure), this operation, through the pressure transmission mechanism within the pipeline network, directly affects the downstream fluid pressure, which is finely controlled by the reinforcement learning controller. This downstream pressure drift caused by upstream pressure adjustment is a predictable result determined by environmental physical characteristics, rather than a failure of the system's own control. However, existing reinforcement learning controllers typically only use the macroscopic pressure average or statistical values ​​within a preset time period as part of their state input space, failing to perceive the crucial mechanical context of "upstream operation occurring." After observing that the downstream pressure deviates from the preset target, the controller, according to its trained, target-oriented reward mechanism, consistently attributes this deviation to a bias that the system needs to correct and automatically adjusts the control valves or pump speed. This leads to a common side effect: to compensate for the upstream physical operation, the reinforcement learning controller performs unnecessary and potentially harmful excessive reverse control, causing artificial oscillations in the control variables.

[0005] In the literature on reinforcement learning techniques involving multivariate continuous decision-making, a complete and effective technical solution is still lacking for handling non-systematic failure-related state drift introduced by identifiable external physical operations. Existing research mainly focuses on improving the ability to model dynamic environments by improving the structure or memory mechanism of state representation networks, such as using state encoders with gated recurrent units or long short-term memory networks to capture temporal dependencies. However, these methods do not systematically solve the signal confusion problem between "external operations of known origin and known physical mechanism" and "internal state variations of the system itself." They attempt to allow neural networks to implicitly learn to distinguish between these two patterns from massive amounts of data, but this often requires massive training data and extremely large model capacity, and in practice, it has been shown to be prone to overfitting to the most common external operation patterns during training. When encountering new operation forces or timings that have not been seen before, misjudgments may still occur.

[0006] It is understood that no technical solution has yet been proposed in this field that can effectively solve the problems of state signal confusion and reinforcement learning control decision failure caused by physical coupling operations from known sources. Summary of the Invention

[0007] The purpose of this invention is to overcome the technical defects of existing reinforcement learning intensive care parameter adjustment systems, which are unable to distinguish between thoracic mechanical effects and pathological deterioration, leading to incorrect drug administration. The invention provides a dynamic adjustment method that enables reinforcement learning agents to acquire mechanical context awareness, thereby internalizing and eliminating non-pharmacological interference caused by changes in thoracic pressure during decision-making.

[0008] To achieve the above objectives, the technical solution of the invention is achieved through the following measures:

[0009] A method for dynamic adjustment of critical care parameters based on reinforcement learning includes: acquiring respiratory mechanics parameter data and continuous invasive arterial blood pressure waveform data of a patient, wherein the respiratory mechanics parameter data includes at least positive end-expiratory pressure; inputting the respiratory mechanics parameter data and the continuous invasive arterial blood pressure waveform data into a mechanics context encoder to generate a mechanics context vector at the current moment; the mechanics context encoder is a dual-channel neural network based on an attention mechanism, and the mechanics context vector represents the real-time influence pattern and intensity of the current respiratory mechanics on blood pressure; performing feature concatenation of the mechanics context vector and a patient physiological state feature vector to obtain an extended state vector; the patient physiological state feature vector is constructed based at least on heart rate, historical blood pressure trends, and the current vasoactive drug infusion rate; inputting the extended state vector into a reinforcement learning agent to output an adjustment action for the vasoactive drug infusion rate; and controlling an infusion pump to perform vasoactive drug infusion according to the adjustment action; wherein the reward function of the reinforcement learning agent uses a dynamic target blood pressure value as a benchmark, and the dynamic target blood pressure value decreases as the positive end-expiratory pressure increases.

[0010] In the above technical solution, generating the mechanical context vector at the current moment includes: acquiring airway pressure waveform data based on the respiratory mechanics parameter data; taking the inspiratory start time of the respiratory cycle as a reference, extracting airway pressure waveform segments from the airway pressure waveform data within a first preset time window, and invasive arterial blood pressure waveform segments from the continuous invasive arterial blood pressure waveform data within a second preset time window aligned with the same time axis, to form waveform dual slices; inputting the waveform dual slices into the dual-channel neural network, with each channel extracting local morphological features of the two waveform segments through a one-dimensional convolutional layer; calculating the causal mapping weights of the airway pressure waveform segments to the invasive arterial blood pressure waveform segments through a cross-channel attention layer to obtain a respiratory-blood pressure coupling matrix; and outputting the mechanical context vector based on the respiratory-blood pressure coupling matrix through self-attention layer aggregation.

[0011] After outputting the adjustment action and before controlling the infusion pump to execute, the method may further include: constructing a linear time-invariant second-order system transfer function model characterizing the dynamic relationship between changes in positive end-expiratory pressure (PEEP) and changes in blood pressure; identifying the model parameters of the transfer function model online using recursive least squares method based on historical respiratory mechanics parameter data and historical blood pressure change data calculated from the continuous invasive arterial blood pressure waveform data; in response to detecting that the current PEEP change exceeds a preset amplitude threshold, predicting the expected blood pressure change caused by the change within a preset future time window using the transfer function model; calculating the comparison result between the actual blood pressure change and the expected blood pressure change within the preset future time window; if it is within a preset tolerance range, determining that the current blood pressure fluctuation is dominated by respiratory mechanics, and applying an inhibitory factor to the adjustment action to obtain a corrected adjustment action; and controlling the infusion pump to execute according to the corrected adjustment action.

[0012] Furthermore, when the change in positive end-expiratory pressure is detected to exceed the preset amplitude threshold, the determination based on the comparison result can be paused within a preset activation protection window immediately following the change, forcibly limiting the change in the infusion rate of the adjustment action to a preset safe change range; after the preset activation protection window ends, the determination and the inhibition factor are then activated again; the duration of the preset activation protection window is preset according to the convergence speed of the online identification using the recursive least squares method, and the preset safe change range is determined according to a preset percentage range of the vasoactive drug infusion rate before the change.

[0013] The method may further include: acquiring historical and current data of the vasoactive drug infusion rate; reconstructing the continuous invasive arterial blood pressure waveform data, the airway pressure waveform data, and the historical and current data of the vasoactive drug infusion rate into a delayed embedding phase space using a delayed embedding method to obtain the system coupling state trajectory; training a single-class classifier using pure mechanical coupling state samples labeled with causal tags to define the boundary of the mechanical coupling kernel in the delayed embedding phase space; acquiring the coordinates of the current system coupling state point in the delayed embedding phase space and inputting them into the single-class classifier to calculate the signed distance to the boundary of the mechanical coupling kernel and generate a coupling health index; adding the coupling health index to the extended state vector or to the penalty weight used to modulate the reward function to increase the penalty weight when the coupling health index is lower than a preset lower threshold.

[0014] The pure mechanical coupling state samples are labeled as follows: historical time period data are screened, which simultaneously meet the following conditions: the positive end-expiratory pressure adjustment is greater than a preset adjustment threshold; within the preset silent period after adjustment, the change in the vasoactive drug infusion rate is less than a preset change threshold; the patient's hemodynamics are stable and there are no additional clinical intervention records within the preset silent period; the system coupling state trajectory data within the historical time period that meets the conditions are labeled as pure mechanical coupling state samples; pathological deterioration period data are screened, which occur during the period when the ventilator parameters are not adjusted and the vasoactive drug infusion rate is actively increased, and the pathological deterioration period data are labeled as negative samples to verify the boundary of the mechanical coupling kernel defined by the single classifier.

[0015] The dynamic target blood pressure value decreasing as end-expiratory positive pressure increases includes: setting a target lower limit of mean arterial pressure in the dynamic target blood pressure value, so that it automatically decreases as the current end-expiratory positive pressure increases based on a preset mapping relationship; the preset mapping relationship is obtained based on a patient-specific intrathoracic pressure transmission coefficient, which is updated online through the quantitative relationship between changes in end-expiratory positive pressure and changes in blood pressure in historical data.

[0016] The patient's physiological state feature vector may further include at least one dynamic preload index among stroke volume variability and pulse pressure variability; obtaining the dynamic preload index includes: extracting systolic blood pressure and pulse pressure stroke by stroke from the continuous invasive arterial blood pressure waveform data, calculating the stroke volume variability or the pulse pressure variability based on the respiratory cycle; adding the dynamic preload index to the patient's physiological state feature vector so that the extended state vector contains volume responsiveness information.

[0017] The reinforcement learning agent may further include, during training: monitoring whether the change in positive end-expiratory pressure or tidal volume exceeds a preset event threshold; in the experience playback cache, assigning a higher sampling priority to sample segments containing events exceeding the preset event threshold than to sample segments not containing such events, wherein the sampling priority is determined comprehensively based on the change in positive end-expiratory pressure and the degree of blood pressure response deviation, wherein the degree of blood pressure response deviation is the deviation between the actual blood pressure change and the expected blood pressure change within the sample segment.

[0018] The reinforcement learning agent can adopt an Actor-Critic architecture, including an Actor network and a Critic network; the Actor network receives the extended state vector and outputs the mean and variance of the adjustment action; the Critic network receives the extended state vector and outputs a state value estimate; the Actor network and the Critic network share the mechanical context vector output by the mechanical context encoder during training.

[0019] The beneficial effects of this invention are as follows:

[0020] By simultaneously inputting airway pressure waveform data and arterial blood pressure waveform data into a biomechanical context encoder and utilizing a cross-channel attention mechanism to learn the causal mapping relationship between them, the output biomechanical context vector enables the reinforcement learning agent to, for the first time, perceive the pattern and intensity of the influence of respiratory mechanics on blood pressure, thereby distinguishing between thoracic biomechanical effects and pathological deterioration. Furthermore, by setting the benchmark of the reward function to a dynamic target blood pressure value that decreases with increasing positive end-expiratory pressure, the tendency of the reinforcement learning agent to erroneously administer vasopressors to compensate for the physical side effects of the ventilator is eliminated at the source of the incentive mechanism. In addition, by introducing a cascade arbitration mechanism and coupling health index monitoring, a causal attribution judgment and safety verification channel independent of the main reinforcement learning control loop is provided, enabling the system to respond correctly and promptly even when genuine pathological deterioration occurs, avoiding delays in treatment due to excessive suppression of drug administration. The overall solution fundamentally solves the high-frequency misjudgment problem of reinforcement learning closed-loop drug administration systems under lung-protective ventilation strategies without adding additional sensors. Attached Figure Description

[0021] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0022] Figure 1This is a schematic diagram of the main control flow of the reinforcement learning-based dynamic adjustment method for critical care parameters according to the present invention.

[0023] Figure 2 This is a schematic diagram of the internal structure and data flow processing of a mechanical context encoder.

[0024] Figure 3 This is a schematic diagram of the control process that includes a cascading arbitration mechanism;

[0025] Figure 4 This is a flowchart illustrating the workflow of the health index monitoring module. Detailed Implementation

[0026] Specific embodiments of the invention will now be described in detail. Although the invention is described in conjunction with these specific embodiments, it should be understood that it is not intended to limit the invention to these specific embodiments. Rather, these embodiments are intended to cover alternative, modified, or equivalent embodiments that may be included within the spirit and scope of the invention as defined by the claims. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. The invention may be practiced without some or all of these specific details.

[0027] When used in conjunction with the terms "comprising," "method comprising," or similar language in this specification and appended claims, the singular forms "a," "some," and "the" include plural references unless the context clearly indicates otherwise. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0028] See Figures 1 to 4 The technical solution of the present invention will be described in detail below with reference to specific embodiments.

[0029] Example 1:

[0030] This embodiment provides a reinforcement learning-based method for dynamic adjustment of critical care parameters, implemented in a standard intensive care unit equipped with an invasive ventilator, invasive arterial blood pressure monitoring, electrocardiogram monitoring, and infusion pump. As a specific application scenario throughout this embodiment, it is assumed that a critically ill patient with acute respiratory distress syndrome due to severe pneumonia is admitted. This patient has already undergone endotracheal intubation and invasive mechanical ventilation, with continuous invasive arterial blood pressure monitoring via a right radial artery catheter, and is simultaneously receiving continuous norepinephrine infusion at a preset rate via a central venous catheter. To implement a lung-protective ventilation strategy, the clinician gradually increases the positive end-expiratory pressure of the ventilator from 5 cmH2O to 15 cmH2O.

[0031] Reference Figure 1As shown, the main control flow of this method begins with the data acquisition step. The patient's respiratory mechanics parameters and continuous invasive arterial blood pressure waveform data are acquired. The respiratory mechanics parameters include at least positive end-expiratory pressure (PEEP). In this embodiment, the acquisition of respiratory mechanics parameters is achieved by establishing a communication connection between a data interface module and the ventilator in the intensive care unit. The data interface module reads the current mechanical ventilation settings and monitoring parameters from the ventilator in real time based on the medical information bus protocol, extracting at least the real-time setting value of PEEP, and also extracting plateau pressure, tidal volume, and inspiratory time as auxiliary features. The acquisition of continuous invasive arterial blood pressure waveform data is achieved through a high-fidelity pressure sensor connected to the patient's arterial catheter. The sensor converts the pressure signal within the artery into an analog electrical signal, which is then digitized by an analog-to-digital converter at a sampling rate of 200Hz to form a continuous data stream. The acquired respiratory mechanics parameters and continuous invasive arterial blood pressure waveform data are timestamped on the time axis to ensure the time alignment accuracy of subsequent coupling analysis. Simultaneously, airway pressure waveform data based on respiratory mechanics parameters is acquired. This airway pressure waveform data is read directly from the ventilator's waveform output interface and is a continuous curve showing the change of airway pressure over time during each respiratory cycle. Historical and current data on the infusion rate of vasoactive drugs are also synchronously acquired through the infusion pump's data interface.

[0032] In the above scenario, the respiratory mechanics parameters acquired by the system from the ventilator include the set values ​​for each step of the positive end-expiratory pressure (PEEP) from 5 to 15 cmH2O. Continuous invasive arterial blood pressure waveform data acquired from the arterial catheter records the complete waveform transition process of systolic blood pressure decreasing from approximately 72 mmHg to approximately 65 mmHg before and after the PEEP change. Airway pressure waveform data demonstrates the process of pressure rising from a baseline of 5 cmH2O to a peak pressure of approximately 20 cmH2O and then falling back when the PEEP is set to 5 cmH2O, as well as the morphological changes in pressure rising from a new baseline of 15 cmH2O to a peak pressure of approximately 30 cmH2O after the PEEP is increased to 15 cmH2O.

[0033] After data acquisition, the respiratory mechanics parameters and the continuous invasive arterial blood pressure waveform data are input into the mechanics context encoder to generate the mechanics context vector for the current moment. (Refer to...) Figure 2 As shown, the mechanical context encoder is constructed as a two-channel neural network model based on an attention mechanism, and its specific workflow is as follows.

[0034] First, using the inspiratory start time of the respiratory cycle as a reference, the starting point of the inspiratory action is automatically detected by analyzing the first derivative of the airway pressure waveform to find the zero-crossing point from negative to positive. Using the detected inspiratory start time as a time anchor point, preset time lengths are extracted both forward and backward to form a first preset time window. Airway pressure waveform segments are extracted within this window. Using the exact same time axis as a reference, invasive arterial blood pressure waveform segments are extracted within a second preset time window. The two windows share the same time axis start point and the same window length, thus forming a set of time-precisely aligned waveform pairs. The window length is set to cover the duration of a complete respiratory cycle, typically 3 to 4 seconds.

[0035] Subsequently, the waveform dual slices are input into a dual-channel neural network for feature extraction. Each channel has its own independent one-dimensional convolutional layer, using multiple trainable convolutional kernels to slide along the time axis. For the channel containing the airway pressure waveform segment, the local morphological features extracted by the one-dimensional convolutional layer include the pressure rise slope, plateau duration, pressure fall rate, and waveform inflection point location. For the channel containing the invasive arterial blood pressure waveform segment, the local morphological features extracted by the one-dimensional convolutional layer include the systolic peak sharpness, the depth and location of the dicrotic notch, pulse width, and waveform contour.

[0036] After local morphological feature extraction, the process proceeds to the cross-channel attention layer. This layer receives feature maps from two convolutional channels, using the airway pressure waveform feature map as the key and value in the attention mechanism, and the arterial blood pressure waveform feature map as the query. Standard attention mechanisms can capture correlations between arbitrary time points, but fail to emphasize physiological causal temporal constraints. To encode physiological delay information known in the art as prior knowledge into the neural network's computation process, this embodiment further constructs a causal time delay perception module. This module does not change the basic attention operation structure, but instead uses the following causal time offset matrix... The original attention score is modulated. This modulation matrix is ​​equivalent to a soft, learnable temporal filter, the center of which is... and bandwidth During training, the values ​​are automatically adjusted to best match the current physiological coupling characteristics of the patient. Therefore, the causal mapping weights obtained through the above modulation process and scaling dot product operation are... This clearly reflects the directional constraint that "changes in the airway pressure waveform occur before the response in the arterial blood pressure waveform," thus accurately characterizing the causal relationship in the coupling matrix. Based on the above causal encoding logic, its complete mathematical expression can be defined by the following attention calculation equation:

[0037]

[0038]

[0039] In the formula, The first in the respiratory-blood pressure coupling matrix Line number The elements of the column represent the time points of the arterial blood pressure waveform. Timing of airway pressure waveform Causal mapping weights; The first in the feature mapping of arterial blood pressure waveform Query vectors at each time point; The first in the characteristic mapping of airway pressure waveform Key vectors at each time point; The dimension of the key vector; and These are the relative time indices of time points in the arterial blood pressure waveform segment and the airway pressure waveform segment, respectively, obtained based on the time axis alignment information within a preset time window, with the unit being the sampling point number; The average physiological delay time parameter is a learnable parameter. The initial value is set based on the population physiological data and is approximately the number of sampling points corresponding to 0.25 seconds. It is fine-tuned during model training using gradient descent. The learnable latency-range bandwidth parameter is initially set to the number of sampling points corresponding to approximately 0.1 seconds, and is also optimized during training. The time difference attenuation weight is used. This design explicitly encodes anatomical and physiological causal constraints into the attention calculation process, so that the attention weight depends not only on the similarity of feature content, but also on whether the time distance is within a reasonable physiological delay range. The calculated respiratory-blood pressure coupling matrix fully records the causal direction, time delay, and distribution information of the influence of respiratory mechanics on blood pressure.

[0040] In the aforementioned application scenario, cross-channel attention layer analysis was performed on the airway pressure waveform segment and the corresponding arterial blood pressure waveform segment as the positive end-expiratory pressure (PEEP) switched from 5 to 15. The calculated respiratory-blood pressure coupling matrix showed that the pressure step change occurring at the end of the airway pressure waveform generated a high-weighted mapping on the arterial blood pressure waveform. This mapping reached its peak after a delay of approximately 0.25 seconds, with the weight concentrated on the systolic blood pressure decrease phase.

[0041] After obtaining the respiratory-blood pressure coupling matrix, information is aggregated through a self-attention layer. The layer adaptively focuses on the most informative region of the coupling matrix, filtering out secondary or noisy correlations, ultimately outputting a fixed-dimensional mechanical context vector. This vector, with a dimension of 64, quantitatively describes the real-time influence pattern and intensity of current respiratory mechanics on blood pressure in a compact numerical encoding form. In the above scenario, this vector encodes the relational information that "the current positive end-expiratory pressure is at a high level, the inhibitory intensity of thoracic mechanics on blood pressure is approximately 7 mmHg, and this inhibitory effect mainly acts on systolic blood pressure with a delay of approximately 0.25 seconds."

[0042] After generating the mechanical context vector, it is concatenated with the patient's physiological state feature vector to obtain an extended state vector. The patient's physiological state feature vector is constructed based on heart rate, historical blood pressure trends, and the current vasoactive drug infusion rate. Heart rate data is obtained from an electrocardiogram monitoring device, and the average of the most recent heartbeat intervals is taken as the current heart rate feature. The historical blood pressure trend is constructed by extracting stroke-by-stroke systolic and diastolic blood pressure values ​​from continuous invasive arterial blood pressure waveform data, and calculating the first-order linear fitting slope of the average arterial pressure within a preset time window (30 seconds). The current vasoactive drug infusion rate is obtained from the data interface of the infusion pump. In a preferred embodiment, the patient's physiological state feature vector also includes at least one dynamic preload index among stroke volume variability and pulse pressure variability. The dynamic preload index is obtained by extracting systolic blood pressure and pulse pressure stroke-by-stroke from continuous invasive arterial blood pressure waveform data using a peak detection algorithm. Pulse pressure is defined as the difference between systolic and diastolic blood pressure. After stroke-by-stroke extraction, statistical calculations are performed based on the complete respiratory cycle. Taking pulse pressure variability as an example, the following calculation formula is constructed:

[0043]

[0044] In the formula, and These are the maximum and minimum values ​​of all pulse pressure values ​​after sliding median filtering within a preset time window based on the start of the respiratory cycle's inspiratory phase.

[0045] Similarly, the calculation of stroke volume variability can be accomplished by analyzing the changes in stroke volume derived from the arterial blood pressure waveform within the same respiratory cycle. In one implementation, a pulse profile analysis algorithm can be used to calculate the stroke volume from the continuous invasive arterial blood pressure waveform data, and then substituted into a calculation formula similar to that for pulse pressure variability.

[0046]

[0047] in, and These represent the maximum and minimum values ​​of the filtered stroke volume within the preset time window. The obtained dynamic preload index values ​​are added as an additional dimension to the patient's physiological state feature vector.

[0048] The feature cascade operation involves concatenating the elements of the mechanical context vector and the patient's physiological state feature vector, merging them into an extended state vector.

[0049] In the above application scenario, when the positive end-expiratory pressure (PEEP) is 15 cmH2O, the heart rate is 102 beats / min, the historical blood pressure trend shows a continuous decline over the past 30 seconds, and the current norepinephrine infusion rate is 0.08 μg / kg / min. The calculated pulse pressure variability is 21%. The mechanical context vector encodes information that the PEEP is expected to lower blood pressure by approximately 7 mmHg. These four pieces of information are concatenated to form an extended state vector.

[0050] The extended state vector is input into the reinforcement learning agent, which outputs a regulating action for the infusion rate of vasoactive drugs. The reinforcement learning agent is constructed using an Actor-Critic architecture. As a preferred implementation, both the Actor network and the Critic network can be implemented as multi-layer fully connected feedforward neural networks. In this embodiment, a three-layer structure is specifically adopted, with 128 neurons in each layer and ReLU activation function. However, those skilled in the art will understand that other layers and neurons can be flexibly adjusted according to actual computing resources and task complexity, or other network structures such as residual networks or recurrent neural networks can be used. The input to the Actor network is the extended state vector, and the output is two values, representing the mean and logarithmic standard deviation of the regulating action, which is modeled as a continuous random variable following a Gaussian distribution. The input to the Critic network is also the extended state vector, and the output is a single scalar value representing the state value estimate. During training, the Actor network and the Critic network share the same mechanical context vector output by the mechanical context encoder, ensuring that both have a completely consistent understanding of the current mechanical context.

[0051] The reward function of the reinforcement learning agent uses a dynamic target blood pressure value as a benchmark, which decreases as the positive end-expiratory pressure (PEEP) increases. Specifically, a lower limit for the target mean arterial pressure is set within the dynamic target blood pressure value, causing it to automatically decrease as the current PEEP increases, based on a preset mapping relationship. This preset mapping relationship is calibrated using a patient-specific intrathoracic pressure transfer coefficient. A set of equations for calculating the dynamic shift of the target blood pressure is constructed as follows:

[0052]

[0053]

[0054] In the formula, For the current moment The dynamic target lower limit of mean arterial pressure; The preset lower limit of the target mean arterial pressure corresponds to the baseline positive end-expiratory pressure. The target value (set to 5 cmH2O) is usually taken as 65 mmHg; This is the positive end-expiratory pressure obtained from respiratory mechanics parameter data at the current moment; The patient-specific intrathoracic pressure transmission coefficient at the current moment; The preset exponentially weighted forgetting factor is set to 0.10; The change in blood pressure measured during the most recently identified positive end-expiratory pressure adjustment event; This represents the change in positive end-expiratory pressure (PEP) during the event. The second equation is automatically executed once after each PEP adjustment event that meets the preset conditions, achieving... Rolling updates.

[0055] In the above application scenario, when the positive end-expiratory pressure increases from 5 cmH2O to 15 cmH2O, the lower limit of the dynamic target blood pressure automatically shifts from 65 mmHg to 58 mmHg (calculated based on a coefficient of 0.7). At this time, the patient's actual blood pressure is 65 mmHg. Although the absolute value has decreased, it is still higher than the current lower limit of the dynamic target of 58 mmHg, and the reward function will not generate a penalty signal.

[0056] Based on the adjustment actions output by the reinforcement learning agent, the infusion pump is controlled to administer vasoactive drugs. In the above scenario, the agent determines that the decrease in blood pressure is mainly caused by the mechanical effect of increased positive end-expiratory pressure, and that the current blood pressure is still within the dynamic target range. Therefore, it outputs the adjustment action of "maintaining the current infusion rate of 0.08 μg / kg / min," and the infusion pump continues to execute at this rate, avoiding erroneous drug over-administration.

[0057] Example 2:

[0058] This embodiment, based on Embodiment 1, further introduces a cascaded arbitration mechanism to enhance the system's safety during the transition period of significant adjustments in positive end-expiratory pressure. (Refer to...) Figure 3 As shown, this mechanism is executed after the output adjustment action and before the control of the infusion pump.

[0059] A linear time-invariant second-order system transfer function model is constructed to characterize the dynamic relationship between changes in positive end-expiratory pressure (PEEP) and changes in blood pressure. The model treats the change in PEEP as the system input and the change in blood pressure as the system output, reflecting the two main inertial stages and one pure time lag stage experienced between changes in the thoracic biomechanical environment and the peripheral arterial blood pressure response. The mathematical expression of the model is as follows:

[0060]

[0061] In the formula, The steady-state gain coefficient represents the final blood pressure change caused by a unit change in positive end-expiratory pressure, which is determined by the patient's chest wall elasticity and blood volume status. The first inertial time constant mainly reflects the dynamic response speed of the venous return process; It is the second inertial time constant, which mainly reflects the dynamic response speed of the pulmonary circulation process; The pure lag time reflects the physical delay between a change in intrathoracic pressure and the onset of a blood pressure response. These are the complex frequency variables in the Laplace transform. All four parameters were obtained online using the recursive least squares method.

[0062] For model parameter calibration, recursive least squares (RLS) is used for online identification. The basic operation of RLS is as follows: whenever a new set of observation data is obtained, the current parameter estimates are corrected, and the correction amount is proportional to the observation prediction error and the gain vector. The system continuously records the occurrence time, adjustment magnitude, and subsequent blood pressure response data for each positive end-expiratory pressure (PEEP) adjustment event, and inputs this data into the RLS formula to track the time-varying characteristics of the parameters.

[0063] When the detected change in the current positive end-expiratory pressure exceeds a preset threshold (2 cmH2O), a cascade arbitration mechanism is triggered. First, a preset activation protection window is entered. The window duration is preset to 15 seconds based on the convergence speed of online identification using recursive least squares, covering approximately 5 respiratory cycles. Within the activation protection window, the judgment based on the comparison results is suspended, and the change in the infusion rate of the adjustment action is forcibly limited to a preset safe range. The preset safe range is determined based on a preset percentage range of the pre-change vasoactive drug infusion rate, taken as ±10% of the pre-change rate.

[0064] After the protection window ends, a transfer function model under the current parameters is used to predict the blood pressure response within a preset future time window (0 to 15 seconds) to obtain the expected blood pressure change. Simultaneously, the actual blood pressure change within this time window is measured. The comparison between the actual and expected blood pressure changes is calculated. If the deviation falls within a preset tolerance range (±20%), the current blood pressure fluctuation is determined to be dominated by respiratory mechanics, and an inhibitory factor of 0.2 is applied to the adjustment action to obtain a corrective adjustment action. If the deviation exceeds the tolerance range, no inhibitory factor is applied.

[0065] In the aforementioned application scenario, when a clinician increases the positive end-expiratory pressure (PEEP) from 5 to 15 in a single move, the change of 10 cmH2O far exceeds the preset threshold. Within the subsequent 15-second protection window, regardless of the suggestions output by the reinforcement learning agent, the magnitude of the adjustment is limited to within ±10% of the current infusion rate of 0.08 μg / kg / min. After the 15-second window expires, the model updates its parameters using approximately five newly acquired complete respiratory cycles, stabilizing the gain parameter at approximately 0.7 mmHg / cmH2O. The transfer function model predicts a blood pressure decrease of approximately 7 mmHg, while actual monitoring shows a decrease of approximately 7.5 mmHg, a deviation of approximately 7%, within the ±20% tolerance range. The system determines this to be mechanically dominant, applying an inhibitory factor of 0.2 to the adjustment action, resulting in a corrective adjustment to maintain the infusion rate, which the infusion pump executes accordingly.

[0066] Example 3:

[0067] This embodiment, based on Embodiment 1 or Embodiment 2, further introduces a coupled health index monitoring module as an independent security verification defense. This module can operate in parallel with or as an alternative to the cascaded arbitration mechanism. (Refer to...) Figure 4 As shown.

[0068] Historical and current data on vasoactive drug infusion rates, continuous invasive arterial blood pressure waveforms, and airway pressure waveforms based on respiratory mechanics parameters were acquired. After strict alignment of the three time series along the time axis, they were reconstructed into a delayed embedding phase space using a delayed embedding method. For each one-dimensional time series, an appropriate delay time parameter was automatically selected using mutual information. Then, the current value of the series and several delayed versions were taken at the current moment to collectively constitute the state point coordinates in the high-dimensional space. The delayed version coordinates of the three series were merged to form the delayed embedding phase space. The continuous movement of state points over time forms the system's coupled state trajectory, providing a panoramic characterization of the state evolution of the coupled dynamic system composed of respiratory mechanics, blood pressure, and drug infusion rate.

[0069] A single-class classifier was trained using purely mechanically coupled state samples labeled with causal tags, defining the boundary of the mechanically coupled kernel in the delayed embedding phase space. Purely mechanically coupled state samples were labeled by screening historical time periods that simultaneously met three conditions: First, the positive end-expiratory pressure adjustment before the time period was greater than a preset adjustment threshold (3 cmH2O); second, within a preset silent period after adjustment (5 minutes), the change in the vasoactive drug infusion rate was less than a preset change threshold (5%); and third, the patient's hemodynamics were stable and there were no additional clinical intervention records within the preset silent period. The system coupling state trajectory data corresponding to historical time periods that met all conditions were labeled as purely mechanically coupled state samples. In addition, data from periods of pathological deterioration were selected as negative samples. This type of data occurred when ventilator parameters were not adjusted and the vasoactive drug infusion rate was actively increased by the clinician, and was used to verify the boundary of the mechanically coupled kernel defined by the single-class classifier.

[0070] The single-class classifier is implemented using a Gaussian kernel-based single-class support vector machine. The following optimization objective equation is constructed in the high-dimensional feature space:

[0071]

[0072]

[0073] In the formula, For the first The coordinate vector of a purely mechanically coupled state sample in the delayed embedding phase space; Indicates passing through the Gaussian kernel function Implicitly defined high-dimensional feature mapping; Let the center vector of the hypersphere be the vector in the high-dimensional feature space. The radius of the hypersphere; For the first The slack variables corresponding to each sample; This represents the total number of samples in purely mechanically coupled states. The preset regularization parameter is set to 0.05, which controls the upper limit of the proportion of outliers allowed in the sample. The kernel coefficients are taken as the reciprocal of the feature dimension. After training, the boundary of this hypersphere is the boundary of the mechanical coupling kernel.

[0074] During actual operation, the coordinates of the current system coupling state points in the delayed embedding phase space are acquired in real time and input into the trained single-class classifier to calculate the signed distance to the boundary of the mechanical coupling kernel. A positive signed distance indicates that the point is inside the kernel, and a negative distance indicates that the point is outside the kernel. This distance is converted into a coupling health index between 0 and 1 using a sigmoid monotonic mapping function. When the coupling health index is lower than a preset lower threshold (0.3), the penalty weight in the reward function is multiplied by a preset amplification factor (5 times), significantly enhancing the intervention power of the reinforcement learning agent under this "alert state"; when the coupling health index recovers to above the threshold, the penalty weight returns to the normal level.

[0075] In the above application scenario, when the positive end-expiratory pressure (PEEP) is 15 cmH2O and the blood pressure is stable at approximately 65 mmHg, the current state point is located inside the mechanical coupling kernel, the signed distance is positive, and the coupling health index is 0.92. If the patient's sepsis worsens several hours later, and the blood pressure drops sharply without changing the PEEP, the system state point escapes the kernel boundary, the signed distance becomes negative, and the coupling health index drops sharply to 0.15. Because this is below the preset lower limit threshold of 0.3, the penalty weight is automatically modulated to 5.0. Any deviation of blood pressure from the target range generates a penalty signal five times that of the normal situation, forcing the reinforcement learning agent to rapidly increase the infusion rate for active intervention, thereby seizing the valuable early treatment window.

[0076] The detailed description of the above embodiments fully illustrates the feasibility, preferred implementation, and technical effects achieved by the present invention. Those skilled in the art can make various modifications and substitutions based on the above description without departing from the principles and spirit of the present invention, and these modifications and substitutions should also be considered within the scope of protection of the present invention.

[0077] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention. The actual method is not limited to this. In conclusion, if those skilled in the art are inspired by this description and design similar methods and embodiments without departing from the spirit of the present invention, they should all fall within the protection scope of the present invention.

Claims

1. A method for dynamic adjustment of critical care parameters based on reinforcement learning, characterized in that, include: Acquire the patient’s respiratory mechanics parameters and continuous invasive arterial blood pressure waveform data, wherein the respiratory mechanics parameters include at least positive end-expiratory pressure; The respiratory mechanics parameter data and the continuous invasive arterial blood pressure waveform data are input into the mechanics context encoder to generate the mechanics context vector at the current moment; the mechanics context encoder is a two-channel neural network based on an attention mechanism, and the mechanics context vector represents the real-time influence pattern and intensity of the current respiratory mechanics on blood pressure. The mechanical context vector is concatenated with the patient physiological state feature vector to obtain an extended state vector; the patient physiological state feature vector is constructed based at least on the historical trends of heart rate and blood pressure and the current vasoactive drug infusion rate. The extended state vector is input into the reinforcement learning agent, which outputs the adjustment action of the vasoactive drug infusion rate; Based on the aforementioned adjustment action, the infusion pump is controlled to perform vasoactive drug infusion; The reward function of the reinforcement learning agent uses a dynamic target blood pressure value as a benchmark, which decreases as the positive end-expiratory pressure increases.

2. The method according to claim 1, characterized in that, The generation of the current mechanical context vector includes: Obtain airway pressure waveform data based on the respiratory mechanics parameters; Based on the inspiratory start time of the respiratory cycle, the airway pressure waveform segment of the airway pressure waveform data within the first preset time window and the invasive arterial blood pressure waveform segment of the continuous invasive arterial blood pressure waveform data within the second preset time window aligned with the same time axis are extracted to form a waveform dual slice. The waveform dual slices are input into the dual-channel neural network, and each channel extracts the local morphological features of the two waveform segments through a one-dimensional convolutional layer. The causal mapping weights of the airway pressure waveform segment to the invasive arterial blood pressure waveform segment are calculated by cross-channel attention layer to obtain the respiratory-blood pressure coupling matrix; Based on the respiratory-blood pressure coupling matrix, the mechanical context vector is output through self-attention layer aggregation.

3. The method according to claim 1, characterized in that, After outputting the aforementioned adjustment action and before controlling the infusion pump to execute it, the following is also included: Construct a transfer function model of a linear time-invariant second-order system that characterizes the dynamic relationship between changes in positive end-expiratory pressure and changes in blood pressure; Based on historical respiratory mechanics parameter data and historical blood pressure variation data calculated from the continuous invasive arterial blood pressure waveform data, the model parameters of the transfer function model are identified online using the recursive least squares method. In response to detecting that the change in the current positive end-expiratory pressure exceeds a preset threshold, the expected change in blood pressure caused by the change within a preset future time window is predicted using the transfer function model. The comparison between the actual blood pressure change and the expected blood pressure change within the preset future time window is calculated. If the result is within the preset tolerance range, it is determined that the current blood pressure fluctuation is dominated by respiratory mechanics, and an inhibitory factor is applied to the regulation action to obtain a corrective regulation action. The infusion pump is controlled to perform the correction and adjustment actions as described above.

4. The method according to claim 3, characterized in that, When the change in positive end-expiratory pressure is detected to exceed the preset amplitude threshold, the method further includes: Within a preset activation protection window immediately following the change, the determination based on the comparison results is paused, and the change in the infusion rate of the adjustment action is forcibly limited to a preset safe change range. After the preset startup protection window ends, the determination and the inhibition factor are then activated; The duration of the preset activation protection window is preset based on the convergence speed of the online identification using the recursive least squares method, and the preset safety change range is determined based on a preset percentage range of the vasoactive drug infusion rate before the change.

5. The method according to claim 2, characterized in that, Also includes: Obtain historical and current data on the infusion rate of the vasoactive drug; The historical and current data of the continuous invasive arterial blood pressure waveform data, the airway pressure waveform data, and the vasoactive drug infusion rate are reconstructed into the delayed embedding phase space using the delayed embedding method to obtain the system coupling state trajectory. A single-class classifier is trained using samples of purely mechanically coupled states labeled by causal tags, and the boundary of the mechanically coupled kernel is defined in the delayed embedding phase space. Obtain the coordinates of the current system coupling state point in the delayed embedding phase space and input them into the single-class classifier. Calculate the signed distance to the boundary of the mechanical coupling kernel and generate the coupling health index. The coupling health index is added to the extended state vector or used to modulate the penalty weight of the reward function to increase the penalty weight when the coupling health index is below a preset lower threshold.

6. The method according to claim 5, characterized in that, The purely mechanical coupling state samples are calibrated in the following manner: Historical time period data is filtered, and the historical time period data simultaneously meets the following conditions: the adjustment range of positive end-expiratory pressure is greater than the preset adjustment threshold; within the preset silent period after adjustment, the change range of vasoactive drug infusion rate is less than the preset change threshold; and the patient's hemodynamics are stable and there are no additional clinical intervention records within the preset silent period. The system coupling state trajectory data within a historical period that meets the conditions is labeled as a pure mechanical coupling state sample. Data on periods of pathological deterioration, which occur when ventilator parameters are not adjusted and the infusion rate of vasoactive drugs is actively increased, are screened and labeled as negative samples to verify the boundary of the mechanical coupling kernel defined by the single-class classifier.

7. The method according to claim 1, characterized in that, The dynamic target blood pressure value decreases as the positive end-expiratory pressure increases, including: Set the target mean arterial pressure lower limit in the dynamic target blood pressure value so that it automatically decreases as the current positive end-expiratory pressure increases based on a preset mapping relationship; The preset mapping relationship is obtained based on the patient-specific intrathoracic pressure transmission coefficient, which is updated online through the quantitative relationship between changes in end-expiratory positive pressure and changes in blood pressure in historical data.

8. The method according to claim 1, characterized in that, The patient physiological state feature vector also includes at least one dynamic preload index among stroke volume variability and pulse pressure variability. Obtaining the dynamic preload index includes: extracting systolic blood pressure and pulse pressure stroke by stroke from the continuous invasive arterial blood pressure waveform data, and calculating the stroke volume variability or the pulse pressure variability based on the respiratory cycle; The dynamic preload index is added to the patient's physiological state feature vector so that the extended state vector contains volume responsiveness information.

9. The method according to claim 1, characterized in that, The reinforcement learning agent, during the training process, also includes: Monitor whether the change in positive end-expiratory pressure or tidal volume exceeds a preset event threshold; In the experience playback cache, sample segments containing events exceeding the preset event threshold are assigned a higher sampling priority than sample segments that do not contain the events. The sampling priority is determined comprehensively based on the magnitude of end-expiratory positive pressure change and the degree of blood pressure response deviation, wherein the degree of blood pressure response deviation is the deviation between the actual blood pressure change and the expected blood pressure change within the sample segment.

10. The method according to claim 1, characterized in that, The reinforcement learning agent adopts an Actor-Critic architecture, including an Actor network and a Critic network; The Actor network receives the extended state vector and outputs the mean and variance of the adjustment action; The Critic network receives the extended state vector and outputs a state value estimate. The Actor network and the Critic network share the mechanical context vector output by the mechanical context encoder during training.

Images