Ultrasonic actuating pump lung control method based on reinforcement learning and fuzzy control

Abstract

The invention discloses an ultrasonic actuation pump lung control method based on reinforcement learning and fuzzy control, and relates to the field of extracorporeal membrane lung oxygenation, and the method comprises the following steps: constructing a fuzzy rule between basic vital signs and pulsation frequency according to expert system knowledge, obtaining a fuzzy controller, and adjusting the average flow; constructing an intelligent agent interaction environment according to flow field conditions of the oxygenation device; constructing a state space and an action space of an intelligent agent control strategy, and keeping the motion of the motor stable; a blood pump control strategy model based on deep reinforcement learning is established and trained, and a blood flow waveform curve and a blood pump control strategy are obtained; and extracting the trained model to perform intelligent control on the motor, and evaluating the control performance of the strategy model. According to the method, the optimal flow waveform is found through deep reinforcement learning, the blood flow field is regulated and controlled, the oxygenation efficiency is remarkably improved under the condition that complex oxygenator design is not needed, different intelligent agents can be obtained according to different oxygenator structures, and high transportability is achieved.

Description

Technical Field

[0001] The invention relates to the field of extracorporeal membrane oxygenation, and in particular to an ultrasonic actuated pump lung control method based on reinforcement learning and fuzzy control. Background Art

[0002] Extracorporeal membrane oxygenation (ECMO) is an extracorporeal life support system, mainly used to treat various acute circulatory and (or) respiratory failures that are ineffective with conventional life support. ECMO is mainly composed of a blood pump and an oxygenator, in which the blood pump pumps venous blood into the oxygenator, and after gas exchange, the blood is pumped into the arterial segment of the circulatory circuit. The integrated blood pump and oxygenator system is usually called an integrated pump-oxygenator or simply "pump lung". This integrated design reduces the number of system components, simplifies operation and management, enhances the portability of the equipment, and reduces the chance of blood contact with air, reducing the risk of infection and blood damage.

[0003] The natural lung has a better gas exchange capacity than the extracorporeal membrane oxygenator, mainly because the natural lung has a large alveolar-capillary contact area of ​​about 100-150 square meters, and the diffusion distance between gas and blood is extremely short, not exceeding 1-2 microns. At rest, the amount of O2 and CO2 that the lung can handle is about 200-250 ml per minute on average for adults, and can reach 10-20 times this amount during exercise, and all use room air. In contrast, the hollow fiber oxygenator membrane area currently used in artificial cardiopulmonary bypass is small (1-4 square meters), and its surface area to blood volume ratio is much lower than that of the natural lung, while the gas diffusion distance is about 10-30 microns (Wnek G, Bowlin G. Lung, Artificial: Basic Principles and Current Applications / William J. Federspiel, Kristie A. Henchir. In: CRC Press; 2008: 1693-1704.), which is an order of magnitude larger than the natural lung. Therefore, it becomes a vital task to enhance the oxygenation efficiency of the artificial lung while ensuring the appropriate size of the instrument.

[0004] In order to improve the oxygenation efficiency of blood, A.Martins Costa et al. adjusted the arrangement of fiber tubes in the oxygenator to explore its effect on the performance of the oxygenator. The researchers changed the number and arrangement of ventilation fiber tubes, measured the oxygenation effect, and verified that the arrangement of fiber tubes does have a significant effect on oxygenation efficiency. However, this method of improving oxygenation efficiency is relatively cumbersome in designing and processing fiber tubes, and it is difficult to adjust the arrangement of fibers to adapt to actual working conditions (CostaAM, Halfwerk FR, Thiel JN, et al. Effect of hollow fiber configuration and replacement on the gas exchange performance of artificial membrane lungs[J]. Journal of Membrane Science, 2023, 680: 121742.). In addition, Ryan A. Orizondo et al. increased the gas transfer of hollow fiber membranes (HFMs) by fiber oscillation, achieving a 40% oxygenation enhancement without significantly increasing hemolysis, but the system structure was relatively complex (Orizondo RA, Gino G, Sultzbach G, et al. Effects of hollow fiber membrane oscillation on an artificial lung [J]. Annals of biomedical engineering, 2018, 46: 762-771.).

[0005] Many existing technical solutions involve indirect regulation of the blood flow field, and it is difficult to dynamically adjust the settings that have been completed. Therefore, exploring technical solutions that directly regulate the blood flow mode by changing the blood pump movement mode is expected to become a potential research field. Among the driving methods of blood pumps, ultrasonic motors show great potential. With their fast and high-precision control capabilities, ultrasonic motors can flexibly adjust blood flow patterns. In addition, the compact design and low noise characteristics of ultrasonic motors make them very suitable for use in clinical environments, reducing patient discomfort. At the same time, its rapid response capability enables the system to flexibly adjust according to real-time monitoring data to dynamically adapt to the patient's physiological needs. This integrated pump-oxygenator system driven by an ultrasonic motor, namely an ultrasonically actuated pump lung, can provide more efficient and personalized blood circulation support for clinical treatment.

[0006] Therefore, technicians in this field are committed to developing an ultrasonic actuated lung pump control method based on reinforcement learning and fuzzy control. Summary of the invention

[0007] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is that the current method of improving oxygenation efficiency is difficult to dynamically adjust, and the device system is relatively complex.

[0008] To achieve the above object, the present invention provides an ultrasonic actuated lung pump control method based on reinforcement learning and fuzzy control, the method comprising the following steps:

[0009] S101: constructing fuzzy rules between basic vital signs and pulsation frequency based on the expert system knowledge, and designing a membership function to obtain a fuzzy controller, wherein the fuzzy controller dynamically adjusts the pump-lung pulsation cycle according to changes in heart rate and mean arterial pressure, thereby adjusting the average flow rate;

[0010] S103: constructing an intelligent agent interaction environment according to the flow field conditions of the oxygenation device, wherein the environment includes a flow field structure and a gas oxygenation model inside the oxygenation device;

[0011] S105: constructing a state space and an action space of the intelligent agent control strategy, wherein the objectives of the intelligent agent control strategy include maintaining stable motor motion, improving oxygenation efficiency, and reducing blood cell damage;

[0012] S107: establishing and training a blood pump control strategy model based on deep reinforcement learning, wherein the strategy model is used to obtain a blood flow waveform curve and a blood pump control strategy that can significantly improve oxygenation efficiency;

[0013] S109: extracting the trained strategy model, performing intelligent control on the motor, and evaluating the control performance of the strategy model.

[0014] Furthermore, in step S101, the input and output of the fuzzy controller are modeled using Gaussian membership functions, including input variables and output variables, the input variables including heart rate and mean arterial pressure, the output variables including pulse frequency, and each input and output variable is divided into three levels: low, normal and high.

[0015] Furthermore, in step S103, a working cycle of the oxygenation device includes a simulated diastolic cycle and a simulated systolic cycle, wherein:

[0016] In the simulated diastolic cycle, the motor drives the slider and the blood pump push plate to move downward, the pressure in the blood cavity decreases, and blood enters the blood cavity through the inlet valve and fills it; oxygen enters the hollow fiber membrane through the gas inlet, and the blood absorbs oxygen and discharges carbon dioxide through the hollow fiber membrane, and the carbon dioxide is finally discharged from the device through the gas outlet;

[0017] During the simulated contraction cycle, the motor drives the slider and the blood pump push plate to move upward, the pressure in the blood cavity increases, and the blood is discharged from the blood outlet after being oxygenated by the hollow fiber membrane, providing oxygenated blood to the human body to maintain normal circulation.

[0018] Furthermore, the step S105 includes the following sub-steps:

[0019] S1051: constructing a state space of the ultrasonically actuated pump lung, wherein the state space is configured to store calculation results of the oxygenation system simulation model on gas transmission efficiency, hemolysis index, and platelet activation level;

[0020] S1052: constructing an action space of the ultrasonically actuated pump lung, wherein the action space automatically adjusts the excitation voltage frequency and duty cycle of the motor to optimize the smooth movement of the motor;

[0021] S1053: Constructing a reward function for the ultrasonically actuated lung pump, wherein the reward function adopts a hierarchical design and includes an immediate reward and a periodic reward.

[0022] Furthermore, the state space is configured as follows:

[0023] s t ={OTE,SO2,HI,PA,x,T,U,I}

[0024]

[0025] Among them, s t is the state space, OTE is the oxygen transfer efficiency, SO2 is the oxygen saturation, HI is the hemolysis index, PA is the platelet activation level, x is the displacement of the blood pump push plate, T is the time required for the blood pump push plate to complete a round-trip periodic motion, that is, the beating cycle, U is the voltage across the motor, and I is the current passing through the motor;

[0026] C oxypre is the amount of oxygen in a unit volume of blood before oxygenation, C oxypost is the amount of oxygen per unit volume of blood after oxygenation, C oxy is the amount of oxygen per unit volume of blood, Hb is the hemoglobin concentration, and pO2 is the oxygen partial pressure.

[0027] Furthermore, the action space is configured as follows:

[0028] a t ={f,d}

[0029] Among them, a t is the action space, f is the excitation voltage frequency of the motor, and d is the excitation voltage duty cycle of the motor.

[0030] Furthermore, the reward function is configured as:

[0031]

[0032] Among them, R total is the reward function, R instant (t) is the instant reward function, reflecting the real-time motion characteristics of the motor, R cycle is a periodic reward function, which is used to evaluate the oxygenation efficiency and blood damage after an oxygenation cycle, and to control the pulsation frequency to meet the output requirements of fuzzy control. γ1 and γ2 are weight coefficients, t is time, and T is a time period. During the training process, in addition to the motor motion state and pulsation cycle, other state information is calculated and obtained by the constructed computational fluid dynamics simulation model. In practical applications, when using the trained strategy, it is necessary to keep the non-instant feedback as a fixed value, that is, keep the blood destructive index and oxygen transfer efficiency OTE as a fixed value, and obtain oxygen saturation, heart rate and mean arterial pressure in real time through the monitor as input state information for the strategy.

[0033] Furthermore, the instant reward function is:

[0034] R instant (t) = λSO2-μ(C energy (t)+C instability (t))

[0035] C energy (t)=U(t)·I(t)

[0036]

[0037] Among them, C energy (t) is the energy consumption in the current time step, U(t) is the voltage through the motor, U(t) is the current through the motor, C instability (t) is the penalty term for the instability of the blood pump push plate motion, which uses the acceleration of the displacement of the blood pump push plate inside the ultrasonically actuated pump lung to reflect the oscillation, and λ and μ are weight coefficients.

[0038] Furthermore, the periodic reward function is:

[0039] R cycle =α·OTE-δ·HI-ε·PA-ζ|T-T0|

[0040] Among them, α, δ, ε, ζ are weight coefficients, ranging from [0, 1], and T0 is the current required time.

[0041] Furthermore, the step S107 includes the following sub-steps:

[0042] S1071: Initializing a policy network and a value network, wherein the policy network generates a control signal according to a current state, wherein the control signal is configured as an action, and the value network evaluates the value of a current state-action pair and outputs a Q value of the action;

[0043] S1072: collecting state data of the motor and flow waveform data corresponding to the state data in real time, and importing the flow waveform data of one cycle into an oxygenation device simulation system to obtain an oxygenation efficiency index;

[0044] S1073: Combining the oxygenation efficiency index and the state data into a complete experience tuple, adding the experience tuple to an experience replay pool, and keeping the experience replay pool updated;

[0045] S1074: Periodically randomly extracting small batches of experience samples from the experience replay pool for training the policy network and the value network;

[0046] S1075: Repeat the data collection and training process, and implement the iterative training process by continuously collecting new experience and optimizing strategies until the strategy model converges.

[0047] Furthermore, when training the policy network and the value network, the policy network is optimized using a policy gradient method, and the value network is optimized using a Bellman error minimization method.

[0048] In a preferred embodiment of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. The present invention uses deep reinforcement learning to find the optimal flow waveform, and improves the oxygenation efficiency by directly regulating the blood flow field. In practical applications, it can be optimized for different working conditions, and the oxygenation efficiency can be significantly improved without the need for complex oxygenator design. It can also be dynamically optimized according to different working conditions. At the same time, it combines fuzzy control, so that the device can automatically adjust the average flow rate of the pump lung output according to the different blood supply needs of the patient in actual use, so as to better adapt to the actual situation of the patient;

[0050] 2. The present invention uses a deep reinforcement learning method, takes the smooth movement of the motor, blood oxygen index and blood destructiveness as optimization goals, and combines the SAC (Soft Actor-Critic) algorithm to train the reinforcement learning model to achieve adaptive and intelligent control of the ultrasonic motor;

[0051] 3. The present invention can be modeled and trained based on different oxygenator structures to obtain different intelligent entities and regulate the actual pumping of the lungs, and has strong portability.

[0052] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the implementation steps of the ultrasonic actuated pump lung control method based on reinforcement learning and fuzzy control according to an embodiment of the present invention;

[0054] Figure 2 is a schematic diagram of the deep reinforcement learning system architecture of an embodiment of the present invention;

[0055] Figure 3 Schematic diagram of the oxygenation device system structure according to an embodiment of the present invention.

[0056] Description of the numbers in the figure:

[0057] 1-blood chamber, 2-hollow fiber membrane, 3-blood inlet, 4-valve, 5-blood outlet, 6-flow data collection and CFD simulation unit, 7-ultrasonic flowmeter, 8-hose, 9-ultrasonic linear motor slider and blood pump push plate, 10-multi-parameter monitor, 11-monitor signal conditioning circuit, 12-control host, 13-MCU controller, 14-drive circuit, 15-housing, 16-ultrasonic linear motor stator, 17-displacement sensor, 18-sensor signal conditioning circuit. DETAILED DESCRIPTION

[0058] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0059] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0060] like Figure 1 , Figure 2As shown, in view of the limitations of the prior art in terms of oxygenation efficiency, the present invention optimizes the blood flow waveform under the guidance of the layer-by-layer oxygenation concept to achieve a significant improvement in oxygenation efficiency. The core of the layer-by-layer oxygenation concept is to use the characteristics of the ultrasonic linear motor to accurately control the micron-level displacement, accurately control the amount of blood involved in oxygenation each time and its residence time in the oxygenator, and ensure that the blood can be discharged in time after being fully oxygenated, thereby maintaining a stable blood flow. At the same time, this method can also ensure that the gas in the fiber tube is most effectively utilized to improve the overall oxygenation performance. The ultrasonic motor's precise management of the amount of blood involved in oxygenation and its residence time is manifested as dynamic regulation of the flow waveform in the oxygenation stage from a macroscopic perspective. This regulation makes the blood flow curve in the oxygenator more in line with the optimal oxygenation requirements, thereby achieving fine control of the blood oxygenation process at the microscopic level, which is highly consistent with the layer-by-layer oxygenation concept.

[0061] The present invention obtains a fuzzy controller by designing fuzzy rules between the patient's basic vital signs and the pulsation frequency, and designs a membership function, so that the controller can dynamically adjust the pump-lung pulsation cycle according to the patient's heart rate and mean arterial pressure changes, and then adjust the average flow rate to better adapt to the dynamic changes of human vital signs.

[0062] An ultrasonically actuated lung pump control method based on reinforcement learning and fuzzy control is proposed in the present invention. The method is based on the idea of ​​layer-by-layer oxygenation. By designing the state space, action space and reward function in the SAC algorithm network model, as well as the Actor and Critic network structure, an entropy regularization coefficient is introduced to enhance the exploratory nature of the strategy, helping the model to better balance exploration and utilization in the learning process, thereby maximizing the expected reward function value.

[0063] like Figure 1 As shown, this embodiment proposes an ultrasonic actuated lung pump control method based on reinforcement learning and fuzzy control. The ultrasonic actuated lung pump control method based on reinforcement learning and fuzzy control specifically includes the following steps.

[0064] Step 1: Based on the expert system knowledge, design the fuzzy rules between the patient's basic vital signs and the pulse frequency, and design the membership function. Both the input and output use the Gaussian membership function.

[0065] The fuzzy system consists of: the input variable is heart rate (HR), the domain is: 50-120bpm, the mean arterial pressure (MAP), the domain is: 50-120mmHg, and the output variable is the pulsation frequency (Frequency, F), the adjustment range is: 70-90bpm. Since the Gaussian membership function has the characteristics of smoothness, adjustability and adaptability, it is suitable for the changes of vital signs. The input and output of the fuzzy controller in the present invention are modeled using Gaussian membership functions, and each input and output variable is divided into three levels: low (Low), normal (Normal) and high (High), which can greatly simplify the calculation. The defuzzification uses the centroid method to obtain the final pulsation frequency, and then the pulsation period T0 can be calculated through the pulsation frequency.

[0066] The Gaussian membership function is in the form of:

[0067]

[0068] It can also be expressed as:

[0069] gaussmf(x,[σ,c]).

[0070] The fuzzy domain of heart rate (HR) is: [50,120] bpm, and the fuzzy subsets can be defined as low, normal, and high. The membership function of heart rate is defined as follows:

[0071]

[0072] The fuzzy domain of mean arterial pressure (MAP) is 50-120 mmHg, and the fuzzy subsets can be defined as low, normal, and high. The membership function of mean arterial pressure is defined as follows:

[0073]

[0074] The fuzzy domain of the output variable beat frequency is 70-90 bpm, and the fuzzy subsets can be defined as low, normal, and high. The beat frequency membership function is defined as follows:

[0075]

[0076] Artificial pump lungs have both oxygenation and blood pumping functions, and require precise control of blood flow to coordinate the full oxygenation and efficient delivery of blood. The design of this fuzzy rule is based on heart rate and mean arterial pressure. Changes in heart rate (HR) and mean arterial pressure (MAP) reflect the patient's hemodynamic state and blood supply demand (flow). An example of fuzzy control design is described as follows: low heart rate and low blood pressure indicate that the circulatory system function is in a low-power state. Reducing the pulsation frequency can avoid excessive shear stress and blood damage, while reducing the uncoordinated burden between the equipment and the patient's cardiovascular system; high heart rate and high blood pressure indicate that the body is in a high metabolic or stress state. Increasing the frequency can enhance the output capacity of the pump, thereby improving hemodynamics under hypertension and supporting the perfusion needs of systemic tissues. The specific fuzzy rules are shown in Table 1.

[0077] Table 1 Fuzzy rules

[0078] Rule Number HR MAP F R1 Low Low Low R2 Low Normal Low R3 Low High Normal R4 Normal Low Low R5 Normal Normal Normal R6 Normal High High R7 High Low Normal R8 High Normal High R9 High High High

[0079] Step 2: According to the flow field conditions of the oxygenation device, construct an intelligent agent interaction environment, which includes the flow field structure and gas oxygenation model inside the oxygenation device.

[0080] like Figure 2 As shown, in this embodiment, one working cycle of the oxygenation device includes a simulated diastolic cycle and a simulated systolic cycle, wherein:

[0081] In the simulated diastolic cycle, the motor drives the slider and the blood pump push plate to move downward, the pressure in the blood cavity decreases, and blood enters the blood cavity through the inlet valve and fills it; oxygen enters the hollow fiber membrane through the gas inlet, and the blood absorbs oxygen and discharges carbon dioxide through the hollow fiber membrane, and the carbon dioxide is finally discharged from the device through the gas outlet;

[0082] During the simulated contraction cycle, the motor drives the slider and the blood pump push plate to move upward, the pressure in the blood cavity increases, and the blood is oxygenated through the hollow fiber membrane and discharged from the blood outlet, providing oxygenated blood to the human body to maintain normal circulation.

[0083] In order to achieve precise control of micron-level displacement, in this embodiment, the motor adopts an ultrasonic linear motor.

[0084] Step 3: Construct the state space and action space of the intelligent agent control strategy. The goals of the intelligent agent control strategy include maintaining stable motor motion, improving oxygenation efficiency, and maintaining low blood cell damage.

[0085] In this step, the state space and action space of the agent's corresponding strategy are determined to maintain stable motor movement, improve oxygenation efficiency, and reduce blood cell damage. The reward function adopts a hierarchical design, in which the immediate reward is intended to reflect the real-time movement characteristics of the motor, and the cycle reward is used to evaluate the oxygenation efficiency after an oxygenation cycle. Figure 2 shown.

[0086] In this embodiment, step 3 also includes the following sub-steps:

[0087] S3.1: Construct the state space of the ultrasonically actuated pump lung, and the state space is configured to store the calculation results of the oxygenation system simulation model for gas transfer efficiency, hemolysis index and platelet activation level.

[0088] In this step, the main program is used to call the oxygenation system simulation model of the computational fluid dynamics software to calculate the oxygen and carbon dioxide transmission efficiency as well as the hemolytic index and platelet activation level, and the output calculation results are imported into the state space.

[0089] In this implementation, the state space takes the following form:

[0090] s t ={OTE,SO2,HI,PA,x,T,U,i}

[0091] Among them, s t is the state space, OTE is the oxygen transfer efficiency, SO2 is the oxygen saturation, HI is the hemolytic index, PA is the platelet activation level, x is the displacement of the blood pump push plate, T is the time required for the blood pump push plate to complete a round-trip periodic motion, that is, the beating cycle, U is the voltage across the motor, and I is the current passing through the motor.

[0092] Oxygen transfer efficiency and blood damage indexes are solved by oxygenation system simulated by computational fluid dynamics. OTE, HI, and PA are calculated by oxygenation system simulation model established by ANSYS finite element analysis software. As periodic indicators, they are updated and recorded after the pump lung completes a beat. The remaining indicators are imported into the state space of MATALB as real-time measurement states.

[0093] Oxygen transfer efficiency is characterized by the amount of gas transferred per unit of blood flow, namely:

[0094]

[0095] Where OTE is the oxygen transfer efficiency, C oxypre is the amount of oxygen in a unit volume of blood before oxygenation, C oxypost is the amount of oxygen per unit volume of blood after oxygenation, C oxyis the amount of oxygen in a unit volume of blood, Hb is the hemoglobin concentration, SO2 is the oxygen saturation, and pO2 is the oxygen partial pressure;

[0096] S3.2: Construct an action space for the ultrasonically actuated pump lung, which automatically adjusts the excitation voltage frequency and duty cycle of the motor to optimize the smooth movement of the motor.

[0097] In this embodiment, the action space is:

[0098] a t ={f,d}

[0099] Among them, a t is the action space, f is the excitation voltage frequency of the motor, and d is the excitation voltage duty cycle of the motor.

[0100] S3.3: Construct a reward function for ultrasonically actuated lung pumping. The reward function adopts a hierarchical design, including immediate rewards and periodic rewards.

[0101] In this embodiment, the reward function is:

[0102]

[0103] Among them, R total is the reward function, R instant (t) is the instant reward function, reflecting the real-time motion characteristics of the motor, R cycle is the periodic reward function, which is used to evaluate the oxygenation efficiency after an oxygenation cycle. γ1 and γ2 are weight coefficients, t is time, and T is a time cycle.

[0104] The immediate reward function is:

[0105] R instant (t) = λSO2-μ(C energy (t)+C instability (t))

[0106] C energy (t)=U(t)·I(t)

[0107]

[0108] Among them, R instant (t) is the immediate reward function, C energy (t) is the energy consumption of the current time step, C instability (t) is the penalty term for the instability of the blood pump push plate motion, and the acceleration of the displacement of the blood pump push plate inside the ultrasonic actuated pump lung is used to reflect the oscillation. U(t) is the voltage passing through the motor, I(t) is the current passing through the motor, and λ,μ are weight coefficients.

[0109] The cycle reward function is:

[0110] R cycle =α·OTE-δ·HI-ε·PA-ζ|T-T0|

[0111] Among them, α, δ, ε, ζ are weight coefficients, which can be adjusted and optimized through training, and the value range is between [0, 1]. At the same time, after the periodic motion ends, the period T of this motion is calculated based on the result of the displacement sensor combined with the internal clock of the controller, and compared with the fuzzy control output result, that is, the current required time T0. If the deviation exceeds 5%, this strategy is directly abandoned.

[0112] Step 4: Establish and train a blood pump control strategy model based on deep reinforcement learning. The strategy model is used to obtain a blood flow waveform curve and a blood pump control strategy that can significantly improve oxygenation efficiency.

[0113] In this embodiment, by establishing and training a blood pump control strategy based on a deep reinforcement learning algorithm, a blood flow waveform curve and a control strategy that can significantly improve oxygenation efficiency can be obtained.

[0114] This embodiment uses the SAC algorithm for strategy training. The SAC algorithm maximizes the cumulative reward while also increasing the exploration capability by maximizing the entropy of the strategy. The algorithm is implemented on the MATLAB platform.

[0115] In this embodiment, step 4 includes a data collection phase and a model training phase, which specifically includes the following sub-steps:

[0116] S4.1: Initialize the policy network and value network. The policy network generates a control signal based on the current state. The control signal is configured as an action. The value network evaluates the value of the current state-action pair and outputs the Q value of the action.

[0117] In this embodiment, the blood pump control strategy model includes a strategy network (Actor) and a value network (Critic). The Actor network is used to generate a control signal (i.e., action) according to the current state, and the Critic network is used to evaluate the value of the current state-action pair and output the Q value of the action. In a complete motor movement cycle, the state of the motor and the corresponding oxygenation efficiency index are recorded in real time.

[0118] S4.2: Collect the state data of the motor and the flow waveform data corresponding to the state data in real time, and import the flow waveform data of one cycle into the oxygenation device simulation system to obtain the oxygenation efficiency index and the blood damage index.

[0119] When collecting real-time operation data of the motor, it is necessary to collect real-time data of a complete motion cycle of the ultrasonic linear motor, and record the state of the ultrasonic linear motor (including speed, voltage, current) and the flow waveform data corresponding to the state in real time.

[0120] The collected flow waveform data of one cycle is imported into the oxygenation device simulation system to obtain the oxygenation efficiency index and blood damage index.

[0121] In the training phase, MATLAB is used to simulate the heart rate and mean arterial pressure under different states, and these input values ​​are passed to the fuzzy controller to output the corresponding beating frequency, thereby determining the beating cycle T0. Under each beating cycle condition, the agent begins training. When the agent interacts with the environment, the replay pool is dynamically updated through the acquired experience data. During the training process, small batches of experience samples are randomly extracted from the replay pool to optimize the Critic and Actor networks. Among them, the Bellman error minimization method is used to optimize the Critic network, and the policy gradient method is used to optimize the decision-making performance of the Actor network. When the agent achieves a better oxygenation effect and motor motion smoothness at the current beating frequency, the input state of the heart rate and mean arterial pressure is adjusted, and the above training process is repeated to further improve the robustness and adaptability of the system.

[0122] S4.3: Combine the oxygenation efficiency index and the state data into a complete experience tuple, add the experience tuple to the experience replay pool, and keep the experience replay pool updated.

[0123] In this step, the oxygenation efficiency index, blood damage index and real-time data are combined into a complete experience tuple, and the newly generated experience tuple is continuously added to the experience replay pool to keep the replay pool updated.

[0124] Meanwhile, during the training phase, whenever the agent interacts with the environment and gains experience, the experience is added to the replay pool.

[0125] S4.4: Regularly randomly extract small batches of experience samples from the experience replay pool for training the policy network and value network.

[0126] During the training phase, small batches of experience samples are randomly drawn from the playback pool periodically to train the Critic network and the Actor network. During the training process, the Actor network is optimized using the policy gradient method, and the Critic network is optimized using the Bellman error minimization method.

[0127] S4.5: Repeat the data collection and training process, and implement the iterative training process by continuously collecting new experience and optimizing strategies until the strategy model converges.

[0128] Repeat the data collection and training process, and realize the iterative process of training by continuously collecting new experience and optimizing strategies until the strategy converges.

[0129] In order to save resource consumption, you can also set an appropriate number of iterations based on the training results and end the training early.

[0130] In this embodiment, when training the policy network and the value network, the policy network is optimized using the policy gradient method, and the value network is optimized using the Bellman error minimization method.

[0131] Step 5: Extract the trained strategy model, perform intelligent control on the motor, and evaluate the control performance of the strategy model.

[0132] The trained strategy model is extracted and compared with the traditional control method. While ensuring the same average flow rate, the comparison includes system operating stability and gas transfer efficiency to evaluate the performance of the strategy.

[0133] like Figure 3 The figure shows the specific structure of the oxygenation device used in this embodiment. The oxygenation device includes a blood chamber 1, a hollow fiber membrane 2, a blood inlet 3, a valve 4, a blood outlet 5, a flow data collection and CFD simulation unit 6, an ultrasonic flow meter 7, a hose 8, an ultrasonic linear motor slider and a blood pump push plate 9, a multi-parameter monitor 10, a monitor signal conditioning circuit 11, a control host 12, an MCU controller 13, a drive circuit 14, a housing 15, an ultrasonic linear motor stator 16, a displacement sensor 17, and a sensor signal conditioning circuit 18.

[0134] The specific contents of a working cycle of the oxygenation device include:

[0135] During the simulated diastole, the motor drives the slider and the blood pump push plate 9 to move downward, the pressure in the blood chamber 1 decreases, and the blood enters the blood chamber 1 through the valve 4 of the blood inlet 3 and fills the blood chamber 1. At the same time, oxygen enters the hollow fiber membrane 2 through the gas inlet, and the blood absorbs oxygen and discharges carbon dioxide through the membrane, and the carbon dioxide is finally discharged from the device through the gas outlet.

[0136] During the simulated contraction, the motor drives the slider and the blood pump push plate 9 to move upward, the pressure in the blood chamber 1 increases, and the blood is oxygenated through the hollow fiber membrane 2 and discharged from the blood outlet 5, providing oxygenated blood to the human body to maintain normal circulation.

[0137] In this embodiment, the establishment of the computational fluid dynamics gas oxygenation model is based on the method proposed by Kaesler A et al. (Kaesler A, Rosen M, Schmitz-Rode T, Steinseifer U, Arens J. Computational Modeling of Oxygen Transfer in Artificial Lungs. Artif Organs. 2018 Aug; 42 (8): 786-799. doi: 10.1111 / aor. 13146. Epub 2018 Jul 24. PMID: 30043394.), which simulates blood as a two-phase system and considers the interaction between hemoglobin and oxygen. The oxygen transfer process in the artificial lung can be more accurately predicted, and the blood damage index is solved by the power law model of blood damage. The values ​​of some model parameters need to be verified and adjusted according to the experimental data of the actual flow field to obtain their accurate values. Finally, combined with ANSYS finite element analysis software, the gas transfer efficiency is numerically calculated and analyzed to evaluate the gas exchange performance and blood destructiveness in the blood circulation system.

[0138] The present invention analyzes gas transmission efficiency based on computational fluid dynamics, and combined with the proposed algorithm, can achieve coupled control of motor motion characteristics and hemodynamic characteristics, thereby obtaining a specific flow waveform to improve oxygenation efficiency. Based on comprehensive consideration of oxygenation efficiency and motor kinematics, the present invention designs the state space, action space and reward function in the SAC algorithm network model, and proposes a control method for ultrasonically actuated pump lungs based on a deep reinforcement learning algorithm, which can achieve precise control of the blood volume and holding time flowing into the oxygenation device, adjust the thickness of the blood side boundary layer, and improve oxygenation efficiency.

[0139] Compared with the prior art, the ultrasonic actuated lung pump control method based on reinforcement learning and fuzzy control provided in the embodiment of the present invention has the following characteristics:

[0140] 1. In view of the fact that the current method of improving oxygenation efficiency is difficult to adjust dynamically and the device system is relatively complex, the present invention uses deep reinforcement learning to find the optimal flow waveform, and improves the oxygenation efficiency by directly regulating the blood flow field. In practical applications, it can be optimized for different working conditions. The present invention can achieve precise control of the blood volume and holding time flowing into the oxygenator by accurately regulating the flow waveform, and adjust the thickness of the boundary layer on the blood side to achieve a layer-by-layer and batch oxygenation process, thereby improving the oxygenation efficiency. Without the need for complex oxygenator design, the oxygenation efficiency can be significantly improved, and it can be dynamically optimized according to different working conditions.

[0141] 2. The working principle of the ultrasonic motor involves complex vibration modes and nonlinear dynamics, which makes its motion control complicated. It is difficult for traditional methods to achieve accurate and smooth control. The present invention uses a deep reinforcement learning method, takes the smooth movement of the motor, blood oxygen index and blood destructiveness as optimization goals, and combines the SAC algorithm to train the reinforcement learning model to achieve intelligent control of the ultrasonic motor. By training an intelligent agent, it can automatically adjust the excitation voltage frequency and duty cycle of the motor according to the real-time state of the motor and blood-related indicators to optimize the smooth movement of the motor, improve the blood oxygen index and reduce the blood destructiveness. The intelligent agent uses the Actor network to generate control actions, the Critic network evaluates the value of these actions, and learns according to the reward function (rewarding smooth movement and blood oxygen improvement, punishing increased blood destructiveness), and finally achieves the control goal, realizing adaptive and intelligent control of the ultrasonic motor.

[0142] 3. In actual applications, the physiological state of patients is in dynamic change, and the blood supply demand will change under different conditions. The present invention monitors the two key physiological parameters of the patient's heart rate (HR) and mean arterial pressure (MAP), and uses the data in the expert system knowledge base to design fuzzy rules to dynamically adjust the average flow rate of the pump lung output to adapt to the patient's changing blood supply needs. By using Gaussian membership functions to model the input and output variables and using the center of gravity method to achieve fuzzy processing, the system can flexibly adjust the pump lung output flow rate. In addition, combined with deep reinforcement learning technology, the system can not only adjust the flow rate in real time according to the patient's status, but also ensure high oxygenation efficiency and low blood damage during the adjustment process, thereby improving the adaptability and effectiveness of the treatment.

[0143] Therefore, the ultrasonic actuated pump lung control method based on reinforcement learning and fuzzy control provided by the embodiment of the present invention can significantly improve the oxygenation efficiency of the artificial pump lung and reduce blood damage, which is of great significance for the research on improving the oxygenation efficiency of the oxygenator. Through the deep reinforcement learning algorithm, precise control can be achieved without modeling the complex physical model of the ultrasonic motor, and the pump body is driven to optimize the flow mode of blood in the oxygenator, thereby improving the exchange efficiency between blood and gas at the phase interface and enhancing the overall efficiency of the oxygenation process. When facing different flow field conditions or different arrangements of hollow fibers, the method of the present invention can be referred to, and the parameters can be appropriately optimized and adjusted to achieve the improvement of the performance of the oxygenator. At the same time, during the training process, considering that it is difficult to feedback blood gas indicators in real time, the present invention can also adopt an offline learning method. In addition, since the experimental items and environmental conditions required for collecting blood gas indicators through actual experiments are relatively harsh, the present invention chooses to adopt the mode of computational fluid dynamics simulation to obtain blood gas indicators. By importing a complete cycle of flow waveform signals generated by the ultrasonic actuated pump lung into the simulation model, the corresponding blood gas indicators can be obtained, which brings great convenience to the actual operation.

[0144] At the same time, the industrial application prospects of the present invention are broad, and it has significant application prospects in improving the performance of artificial heart-lung machines. With the aging of the global population and the advancement of medical technology, the demand for efficient and safe extracorporeal life support systems is growing. The present invention can optimize the flow of blood in the oxygenator through a reinforcement learning algorithm in a limited device space, without physically modifying the internal structure of the oxygenator or the arrangement of hollow fibers. At the same time, the present invention can perform modeling training based on different oxygenator structures, thereby obtaining different intelligent agents, regulating the actual pump lung, and has strong portability.

[0145] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. An ultrasonic actuated lung pump control method based on reinforcement learning and fuzzy control, characterized in that: The method comprises the following steps: S101: constructing fuzzy rules between basic vital signs and pulsation frequency based on the expert system knowledge, and designing a membership function to obtain a fuzzy controller, wherein the fuzzy controller dynamically adjusts the pump-lung pulsation cycle according to changes in heart rate and mean arterial pressure, thereby adjusting the average flow rate; S103: constructing an intelligent agent interaction environment according to the flow field conditions of the oxygenation device, wherein the environment includes a flow field structure and a gas oxygenation model inside the oxygenation device; S105: constructing a state space and an action space of the intelligent agent control strategy, wherein the objectives of the intelligent agent control strategy include maintaining stable motor motion, improving oxygenation efficiency, and reducing blood cell damage; S107: establishing and training a blood pump control strategy model based on deep reinforcement learning, wherein the strategy model is used to obtain a blood flow waveform curve and a blood pump control strategy that can significantly improve oxygenation efficiency; S109: extracting the trained strategy model, performing intelligent control on the motor, and evaluating the control performance of the strategy model.

2. The method according to claim 1, characterized in that In step S103, a working cycle of the oxygenation device includes a simulated diastolic cycle and a simulated systolic cycle, wherein: In the simulated diastolic cycle, the motor drives the slider and the blood pump push plate to move downward, the pressure in the blood cavity decreases, and blood enters the blood cavity through the inlet valve and fills it; oxygen enters the hollow fiber membrane through the gas inlet, and the blood absorbs oxygen and discharges carbon dioxide through the hollow fiber membrane, and the carbon dioxide is finally discharged from the device through the gas outlet; During the simulated contraction cycle, the motor drives the slider and the blood pump push plate to move upward, the pressure in the blood cavity increases, and the blood is discharged from the blood outlet after being oxygenated by the hollow fiber membrane, providing oxygenated blood to the human body to maintain normal circulation.

3. The method according to claim 2, characterized in that The step S105 includes the following sub-steps: S1051: constructing a state space of the ultrasonically actuated pump lung, wherein the state space is configured to store calculation results of the oxygenation system simulation model on gas transmission efficiency, hemolysis index, and platelet activation level; S1052: constructing an action space of the ultrasonically actuated pump lung, wherein the action space automatically adjusts the excitation voltage frequency and duty cycle of the motor to optimize the smooth movement of the motor; S1053: Constructing a reward function for the ultrasonically actuated lung pump, wherein the reward function adopts a hierarchical design and includes an immediate reward and a periodic reward.

4. The method according to claim 3, characterized in that The state space configuration is: s t ={OTE,SO2,HI,PA,x,T,U,I} Among them, s t is the state space, OTE is the oxygen transfer efficiency, SO2 is the oxygen saturation, HI is the hemolysis index, PA is the platelet activation level, x is the displacement of the blood pump push plate, T is the time required for the blood pump push plate to complete a round-trip periodic motion, that is, the beating cycle, U is the voltage across the motor, and I is the current passing through the motor; C oxypre is the amount of oxygen in a unit volume of blood before oxygenation, C oxypost is the amount of oxygen per unit volume of blood after oxygenation, C oxy is the amount of oxygen per unit volume of blood, Hb is the hemoglobin concentration, and pO2 is the oxygen partial pressure.

5. The method according to claim 4, characterized in that The action space configuration is: a t ={f,d} Among them, a t is the action space, f is the excitation voltage frequency of the motor, and d is the excitation voltage duty cycle of the motor.

6. The method according to claim 5, characterized in that The reward function is configured as: Among them, R total is the reward function, R instant (t) is the instant reward function, reflecting the real-time motion characteristics of the motor, R cycle is the periodic reward function, which is used to evaluate the oxygenation efficiency after an oxygenation cycle. γ1 and γ2 are weight coefficients, t is time, and T is a time cycle.

7. The method according to claim 6, characterized in that The immediate reward function is: R instant (t)=λSO2-μ(C energy (t)+C instability (t)) C energy (t)=U(t)·I(t) Among them, C energy (t) is the energy consumption in the current time step, U(t) is the voltage through the motor, I(t) is the current through the motor, C instability (t) is the penalty term for the instability of the blood pump push plate motion, which uses the acceleration of the displacement of the blood pump push plate inside the ultrasonically actuated pump lung to reflect the oscillation, and λ and μ are weight coefficients.

8. The method according to claim 7, characterized in that The periodic reward function is: R cycle =α·OTE-δ·HI-ε·PA-ζ|T-T0| Among them, α, δ, ε, ζ are weight coefficients, and their value range is between [0, 1], and T0 is the current required time.

9. The method according to claim 8, characterized in that The step S107 includes the following sub-steps: S1071: Initializing a policy network and a value network, wherein the policy network generates a control signal according to a current state, wherein the control signal is configured as an action, and the value network evaluates the value of a current state-action pair and outputs a Q value of the action; S1072: collecting state data of the motor and flow waveform data corresponding to the state data in real time, and importing the flow waveform data of one cycle into an oxygenation device simulation system to obtain an oxygenation efficiency index; S1073: Combining the oxygenation efficiency index and the state data into a complete experience tuple, adding the experience tuple to an experience replay pool, and keeping the experience replay pool updated; S1074: Periodically randomly extracting small batches of experience samples from the experience replay pool for training the policy network and the value network; S1075: Repeat the data collection and training process, and implement the iterative training process by continuously collecting new experience and optimizing strategies until the strategy model converges.

10. The method according to claim 9, characterized in that When training the policy network and the value network, the policy network is optimized using a policy gradient method, and the value network is optimized using a Bellman error minimization method.

Images