Oxygen mask oxygen supply control method and system based on deep reinforcement learning

Abstract

The invention discloses an oxygen mask oxygen supply control method and system based on deep reinforcement learning. The method comprises the steps that a control instruction a is generated according to the output state s of an oxygen mask oxygen supply device through a neural network, and an oxygen mask control device is regulated and controlled according to the control instruction a; wherein the neural network training step comprises the following steps: S1, determining an output state s of an oxygen mask oxygen supply device; s2, based on the output state s of the oxygen mask oxygen supply device, selecting an action a according to a 1-epsilon greedy strategy, executing the action a to obtain a corresponding reward r and a state s' at the next moment, and obtaining a tetrad (s, a, r, s'); and S3, randomly extracting tetrad data, and training the neural network by using a deep reinforcement learning algorithm until an iteration requirement is met. The control parameters can be dynamically adjusted, limitation of a fixed controller is eliminated, and more flexible and personalized oxygen supply control is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of oxygen supply control, and more specifically to an oxygen mask oxygen supply control method and control system based on deep reinforcement learning. Background Art

[0002] As an emergency life-saving equipment consisting of an oxygen source, an oxygen regulator, and a breathing mask, the oxygen mask oxygen supply system can provide life safety protection for special groups of people, and is therefore widely used in aerospace, medical rescue, plateau scientific research, and other fields. With the rapid development of aviation, aerospace, and medical fields, the performance and reliability requirements of oxygen mask oxygen supply systems are becoming increasingly higher.

[0003] As the core component of the oxygen mask oxygen supply system, the oxygen regulator plays a decisive role in oxygen supply performance by monitoring and analyzing the user's physiological condition and status, and providing the human body with oxygen or oxygen-enriched air of a certain concentration, pressure and flow rate according to prescribed physiological requirements.

[0004] At present, the existing oxygen mask oxygen supply control system usually adopts traditional control methods, such as PID control, fuzzy control, and active disturbance rejection control (ADRC). These methods often rely on fixed parameter settings and are difficult to adapt to the changes in breathing needs of different users or in different environments, resulting in control lag and unstable oxygen supply effect. With the development of artificial intelligence and machine learning technology, although a large number of intelligent control methods based on learning have emerged, they are only used as compensation items and it is difficult to get rid of the problem of optimizing fixed parameter values.

[0005] Therefore, how to overcome the above-mentioned shortcomings is a problem that those skilled in the art need to solve urgently. Summary of the invention

[0006] In view of this, in order to improve the intelligence level of pressure regulation and get rid of the limitations of the optimal solution of fixed parameters, this application introduces deep reinforcement learning and provides an oxygen mask oxygen supply control method and control system based on deep reinforcement learning.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] On the one hand, the present application discloses an oxygen mask oxygen supply control method based on deep reinforcement learning, including using a neural network to generate a control instruction a according to an output state s of an oxygen mask oxygen supply device, and regulating the oxygen mask control device according to the control instruction a;

[0009] The neural network is trained through the following steps:

[0010] S1, determine the output state s of the oxygen mask oxygen supply device, including: the error between the expected signal and the actual signal of the pressure difference between the inside and outside of the mask, the error derivative and the respiratory flow;

[0011] S2, based on the output state s of the oxygen mask oxygen supply device, select action a according to the 1-ε greedy strategy, execute action a to obtain the corresponding reward r and the state s' at the next moment, and obtain the four-tuple (s, a, r, s');

[0012] S3. Randomly extract quadruple data and use the deep reinforcement learning algorithm to train the neural network until the iteration requirements are met.

[0013] Preferably, the oxygen mask oxygen supply device comprises a proportional solenoid valve and a breathing structure, and the proportional solenoid valve realizes oxygen supply regulation of the oxygen mask control device according to the control instruction a.

[0014] Preferably, the breathing structure includes a simulated lung, a breathing cavity and an exhalation valve. The error and error derivative between the expected signal and the actual signal of the pressure difference between the inside and outside of the mask are obtained through the breathing cavity, and the breathing flow is obtained according to the simulated lung; the expression is:

[0015]

[0016] In the formula, s1(k) represents the first state at time k, s2(k) represents the second state at time k, s3(k) represents the third state at time k, and y r represents the expected signal of the pressure difference between the inside and outside of the mask, y(k) represents the actual signal of the pressure difference between the inside and outside of the mask at time k, y(k)-y(k-1) represents the error of the actual signal of the pressure difference between the inside and outside of the mask at time k minus the actual signal at the previous moment, which reflects the change trend of the error, Q b (k) represents respiratory flow.

[0017] Preferably, the expression of the 1-ε greedy strategy is:

[0018]

[0019] Where ε is the exploration parameter and A(s) represents the action set.

[0020] Preferably, when executing action a, the corresponding reward is obtained according to the following reward function:

[0021]

[0022] Wherein, s1 represents the first state, s2 represents the second state, u represents the evaluation of energy consumption, and p and q represent weight coefficients.

[0023] Preferably, when executing action a, the corresponding reward is obtained according to the following reward function:

[0024] r(k)=-|s1|-m|s2|

[0025] In the formula, s1 represents the first state, s2 represents the second state, and m represents the adjustment coefficient of the second state.

[0026] Preferably, in each loop iteration, the termination condition is,

[0027] Achieve the goal: the pressure difference between the inside and outside of the mask is controlled below 0.3KPa for two consecutive breathing cycles;

[0028] Stop loss in time: 500 consecutive pressure difference data are greater than 10KPa;

[0029] Or time limit: The training duration of one cycle has reached 30s and one of the above conditions is not met.

[0030] Preferably, the deep reinforcement learning algorithm is a DDQN algorithm or a DDPG algorithm.

[0031] Preferably, the quaternary group includes simulation data and real-time control data.

[0032] On the other hand, the present application provides an oxygen mask oxygen supply control system based on deep reinforcement learning, comprising an oxygen supply controller and an oxygen mask, wherein the output end of the oxygen supply controller is connected to the control end of the oxygen mask, and the output end of the oxygen mask is connected to the input end of the oxygen supply controller;

[0033] The oxygen supply controller internally stores the deep reinforcement learning algorithm in the oxygen mask oxygen supply control method based on deep reinforcement learning described in any one of claims 1-8; it is used to generate a control instruction a according to the output state s of the oxygen mask oxygen supply device using the deep reinforcement learning algorithm, and regulate the control end of the oxygen mask according to the control instruction a.

[0034] Preferably, the oxygen mask oxygen supply device comprises a proportional solenoid valve and a breathing structure, and the breathing structure comprises a simulated lung, a breathing chamber and an exhalation valve.

[0035] This application uses a deep reinforcement learning algorithm to train a neural network, allowing it to interact and learn with the oxygen mask oxygen supply system as a dynamic controller, thereby directly generating a control signal to achieve pressure regulation. Compared with the prior art, the oxygen mask oxygen supply control method and control system based on deep reinforcement learning disclosed in the present invention can dynamically adjust the control parameters, thereby overcoming the shortcomings of traditional control methods in adaptability and achieving more flexible and personalized oxygen supply control; and can analyze the user's physiological state in real time, quickly generate optimal control instructions, and significantly improve the real-time and accuracy of oxygen supply control;

[0036] At the same time, by learning the user's breathing pattern through a neural network, the oxygen flow and pressure can be accurately adjusted according to actual needs, thereby optimizing the oxygen supply efficiency and reducing unnecessary oxygen consumption.

[0037] In addition, this application can more accurately match the user's breathing needs, providing a comfortable oxygen supply experience while also improving the safety of oxygen supply;

[0038] Due to the introduction of deep reinforcement learning algorithms, the performance of the oxygen mask oxygen supply control system has been significantly improved. It can not only be applied to traditional aerospace and medical rescue fields, but also can be expanded to more scenarios that require precise oxygen supply control, such as plateau operations, diving operations, and military training in special environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0040] Figure 1 A schematic diagram of a deep reinforcement learning neural network controlling an oxygen mask oxygen supply control device according to the present invention;

[0041] Figure 2 : is a comparison diagram of the effect of the reward function after improvement and the reward function before improvement of the present invention; wherein a is the effect diagram before improvement, and b is the effect diagram after improvement;

[0042] Figure 3 This is a schematic diagram of the deep reinforcement learning neural network training process of the present invention;

[0043] Figure 4 The effect comparison diagram before and after adjusting the simulation step length of the present invention is shown in Figure a, which is the effect diagram before adjustment and Figure b, which is the effect diagram after adjustment;

[0044] Figure 5 This is a schematic diagram of the DDPG training reward value of the present invention;

[0045] Figure 6 The test result diagram of the DDPG controller of the present invention, a is the pressure difference response diagram, and b is the control input diagram;

[0046] Figure 7 The pressure regulation effect diagram of the present invention in the long-distance running scenario, a is the pressure difference response diagram, and b is the control input diagram;

[0047] Figure 8 This is a diagram of the pressure regulation effect in the sleep scene of the present invention, a is the pressure difference response diagram, and b is the control input diagram;

[0048] Fig. 9 The pressure regulation effect of the present invention in the diving scene, a is the pressure difference response diagram, and b is the control input diagram;

[0049] Fig.10 This is the pressure regulation effect of the present invention in the plateau scene, a is the pressure difference response diagram, and b is the control input diagram. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] In order to solve the problem of optimizing the values ​​of fixed parameters, to further improve the intelligence level of pressure regulation, and to get rid of the limitations of the optimal solution of fixed parameters, this application introduces a neural network trained by deep reinforcement learning, which is used to generate a control instruction a according to the output state s of the oxygen mask oxygen supply device, and to regulate the oxygen mask control device according to the control instruction a.

[0052] This application will use a neural network trained with a deep reinforcement learning algorithm as a dynamic controller to interact with the oxygen mask oxygen supply system to directly generate a control signal to complete pressure regulation.

[0053] Embodiment 1:

[0054] Figure 1 This is a schematic diagram of a deep reinforcement learning neural network controlling an oxygen mask supply control device. In this section, a deep reinforcement learning neural network is used as a pressure regulation controller for an oxygen mask supply system, where the oxygen mask supply system is regarded as the environment, and the control module embedded with a deep reinforcement learning algorithm is equivalent to an intelligent agent. The state value and reward value are generated through continuous interaction between the two, thereby training the network to the optimal state.

[0055] In this embodiment, the oxygen mask oxygen supply device includes a proportional solenoid valve and a breathing structure, and the proportional solenoid valve realizes the oxygen supply regulation of the oxygen mask control device according to the control instruction a.

[0056] The neural network is trained through the following steps:

[0057] S1, determine the output state s of the oxygen mask oxygen supply device,

[0058] The definition of the state in this embodiment is determined by the control target of the oxygen mask oxygen supply system, specifically including: the error between the expected signal and the actual signal of the pressure difference between the inside and outside of the mask, the error derivative and the respiratory flow;

[0059] In one embodiment, the breathing structure includes a simulated lung, a breathing cavity and an exhalation valve. The error and error derivative between the expected signal and the actual signal of the pressure difference between the inside and outside of the mask are obtained through the breathing cavity, and the breathing flow is obtained according to the simulated lung; the expression is:

[0060]

[0061] In the formula, s1(k) represents the first state at time k, which is the error between the expected signal and the actual signal of the pressure difference between the inside and outside of the mask, reflecting the quality of the current state; s2(k) represents the second state at time k, which is the error derivative and reflects the trend of the error change; s3(k) represents the third state at time k, which is the respiratory flow of the simulated lung, Q b (k) represents the respiratory flow, y r y(k) represents the expected signal of the pressure difference between the inside and outside of the mask, y(k) represents the actual signal of the pressure difference between the inside and outside of the mask at time k; y(k)-y(k-1) represents the pressure difference at one moment minus the error at the previous moment, which reflects the changing trend of the error and does not need to be adjusted.

[0062] S2, based on the output state s of the oxygen mask oxygen supply device, select action a according to the 1-ε greedy strategy, execute action a to obtain the corresponding reward r and the state s' at the next moment, and obtain the four-tuple (s, a, r, s');

[0063] In one embodiment, the expression of the 1-ε greedy strategy is:

[0064]

[0065] In the formula, ε is the exploration parameter, which usually ranges from 0 to 1. A larger ε value increases the possibility of exploration, while a smaller ε value means a greater tendency to choose the currently known optimal action. This can achieve a balance between utilization and exploration.

[0066] A(s) represents the action set, that is, all the actions that the agent can choose in the current state s, and |A(s)| represents the number of actions in the action set A(s). This value is used to calculate the uniform probability when exploring actions. The meaning of this strategy is to select the action that maximizes the Q value with a probability of 1-ε, otherwise a random action is selected. The main purpose is to ensure that the agent has the ability to explore the global situation, try as many actions as possible to evaluate the value, and avoid falling into the local optimum.

[0067] In one embodiment, since the operation of the simulated lung does not change with the change of the system state, but only with time, it can be judged whether the individual is in the inhalation or exhalation stage based on this, so as to decide whether oxygen supply is needed, avoiding the problem of continuous oxygen supply when the respiratory pressure difference is small. The reward function is designed based on the above analysis, and its purpose is to allow the intelligent agent to choose the best control amount, on the one hand to control the pressure difference near the expected value, and on the other hand to save gas flow as much as possible. The following rules are obtained:

[0068] (a) When s1 is smaller, it means that the pressure difference between the inside and outside of the mask is closer to the expected value, and the reward value is larger;

[0069] (b) When s1s2≤0, it means that the pressure difference between the inside and outside of the mask changes towards the expected value, and the reward value is greater. Therefore, the reward function at time k is expressed as:

[0070]

[0071] In the formula, s1 represents the first state, s2 represents the second state, u represents the evaluation of energy consumption, p and q represent weight coefficients; the sign() function is used to obtain the sign of a number. Specifically: if the value is positive, the sign() function returns 1; if the value is 0, the sign() function returns 0; if the value is negative, the sign() function returns -1.

[0072] However, this function contains multiple conditional terms and multiple absolute value operations, plus additional weight coefficients (such as p, q) and different bias terms (such as 10u), which makes the calculation complicated and requires many considerations, increasing the computational burden of training. At the same time, the redundant terms may lead to greater noise during training, making it difficult for DDPG to capture the main optimization direction.

[0073] Therefore, in order to further optimize the above technical solution, this application proposes a reward function to overcome the above problems, which is expressed as:

[0074] r(k)=-|s1|-m|s2|

[0075] Wherein, m is the adjustment coefficient of the second state S2. In one embodiment, m is 0.5.

[0076] This embodiment considers the error and change rate to design the reward function, which can more accurately reflect the optimization goal of the control system. Figure 2 ,Depend on Figure 2 It can be seen that the negative pressure inside the oxygen mask is significantly reduced, and the effect is remarkable.

[0077] Although this application no longer directly considers energy consumption, by simplifying the control error and rate of change, it can indirectly affect energy use, allowing the algorithm to achieve more economical control under a stable strategy.

[0078] S3. Randomly extract quadruple data and use the deep reinforcement learning algorithm to train the neural network until the iteration requirements are met.

[0079] The purpose of training is to allow the agent to continuously explore and reach the target state as quickly as possible. In each cycle, if the agent performs poorly for a long time or reaches the target state too quickly, the current cycle is terminated and the next cycle iteration is started. The following termination conditions are designed for this:

[0080] Achieve the goal: the pressure difference between the inside and outside of the mask is controlled below 0.3KPa for two consecutive breathing cycles;

[0081] Stop loss in time: 500 consecutive pressure difference data are greater than 10KPa;

[0082] Or time limit: The training duration of one cycle has reached 30s and one of the above conditions is not met.

[0083] In one embodiment, the deep reinforcement learning algorithm is a DDQN algorithm or a DDPG algorithm.

[0084] (1) When using the DDQN algorithm to design a controller, the action space of the DDQN must first be divided, the upper and lower values ​​must be determined, and then discretized. From the outlet flow characteristics of the proportional solenoid valve, it can be seen that there is a certain dead zone characteristic inside the solenoid valve. When the applied PWM duty cycle reaches about 0.65, the valve can only open. After that, as the duty cycle continues to increase, the outlet flow increases linearly until the maximum control signal is 1. Therefore, the discretization design is as follows:

[0085] u∈(u min ,u min +h,...,u max )

[0086] Where h = 0.001 is the sampling interval, u min =0.65 and u max =1 represents the minimum and maximum values ​​of the control signal respectively, so the dimension of the action space is n=(u max -u min ) / h=350.

[0087] During the data accumulation phase, the controller generates actions through the 1-ε strategy and acts on the proportional solenoid valve to provide gas to the oxygen mask, record the system status and reward value, and save the four-tuple (s, a, r, s′) to the experience pool until the number reaches the minimum number of data to train the neural network. In the training phase, each iteration randomly extracts a number of data from the experience pool to train the estimated neural network, and updates the target network parameters every certain number of steps.

[0088] In the testing phase, the parameters of the trained DDQN network are saved and no longer changed, and the parameter ε in the action selection 1-ε strategy is set to 0, the randomness of the action selection is cancelled, and it is completely determined by DDQN. The pressure changes during breathing are simulated and observed to verify the quality of the controller.

[0089] (2) When the DDPG algorithm is used to design the controller, DDPG directly calculates the control quantity based on the current state. There is no need to discretize the action space like DDQN. It only needs to perform limiting processing, that is, the upper and lower bounds of the action space. From the above, it can be seen that the maximum and minimum values ​​of the action are 1 and 0.65 respectively. Then:

[0090] a t =π t (s t θ μ ) t ∈[0.65,1]

[0091] In the formula, π t For the strategy network.

[0092] In this embodiment, in order to ensure that the intelligent agent has the ability of global exploration, random noise is introduced in the training process, the decision-making process of the action is changed from deterministic to a random process, and the value of the action is then sampled from the random process.

[0093] Different from DDQN, during the training and testing process of DDPG, the policy network obtains the PWM duty cycle based on the current respiratory pressure difference, pressure difference change rate and respiratory flow rate, and calculates the Q value with the state and control amount as a vector, so as to update the policy network parameters by using the gradient ascent algorithm, and then calculates the Q value at the next moment through the target network, and further uses the gradient descent algorithm to update the value network parameters, and then uses the soft update algorithm to update the target network. Since the algorithm idea is similar to DDQN, it will not be repeated here.

[0094] In one embodiment, if Figure 3 As shown, the quaternary group includes simulation data and real-time control data, aiming to improve the training effect and optimize the performance of the network.

[0095] In this embodiment, the data source of the experience pool includes not only the simulated environment data, but also the real-time control data, thereby providing a richer data sample. Secondly, while randomly extracting environmental samples from the experience pool, a certain number of samples are also extracted from the real-time control data, so that the neural network can be trained in a closer to real environment. This improvement makes the training process more diversified, which helps to improve the performance of the network in practical applications.

[0096] In a specific embodiment,

[0097] Training phase: Use simulation environment data for offline pre-training;

[0098] Testing phase: Real-time data will be replayed into the experience pool, and the experience pool data is used to update the policy network and value network online. The control action is obtained by the real-time updated policy network. Please refer to the following table for the specific process:

[0099]

[0100] The incremental real-time data update mechanism provided by this application makes the data in the experience pool more dynamic and more in line with the data distribution in actual application scenarios. In addition, the added feedback loop can update the weight of the policy network after each new policy output is obtained, so that the model can continue to learn and optimize, thereby improving the adaptability and stability of the model in practical applications.

[0101] This improvement has important practical significance for systems that require frequent decision-making and real-time response, especially oxygen mask control systems in flight control.

[0102] In one embodiment, in order to speed up the simulation process and improve the convergence speed of the algorithm, the present application adjusts the simulation step size, that is, updates are performed every 100 simulation steps, and the update frequency is reduced to 0.1 seconds. The purpose is to maintain a sufficient update frequency while greatly reducing the amount of calculation, thereby improving training efficiency.

[0103] At the same time, by adjusting the simulation step size, the oscillation caused by the untimely response to the frequent changes in the inhalation phase due to the update of each simulation step size can be eliminated;

[0104] In one embodiment, referring to Figure 4 , simulation time 30s, actual running time 300s, at this time the oscillation in the inhalation stage is serious; after adjusting and updating the simulation step size according to the method of the present application, the simulation time is 30s, the actual running time is 20s, and the oscillation in the inhalation stage is eliminated.

[0105] Embodiment 2:

[0106] This embodiment provides an oxygen mask oxygen supply control system based on deep reinforcement learning, including an oxygen supply controller and an oxygen mask, wherein the output end of the oxygen supply controller is connected to the control end of the oxygen mask, and the output end of the oxygen mask is connected to the input end of the oxygen supply controller;

[0107] The oxygen supply controller internally stores the deep reinforcement learning algorithm in the oxygen mask oxygen supply control method based on deep reinforcement learning as described above; it is used to generate a control instruction a according to the output state s of the oxygen mask oxygen supply device using the deep reinforcement learning algorithm, and regulate the control end of the oxygen mask according to the control instruction a.

[0108] Preferably, the oxygen mask oxygen supply device comprises a proportional solenoid valve and a breathing structure, and the breathing structure comprises a simulated lung, a breathing chamber and an exhalation valve.

[0109] Effectiveness analysis, taking the DDPG algorithm as an example, first train the DDPG controller, set the number of iterations to 300, and the reward value is as follows Figure 5 As shown in the figure, the reward quickly stabilizes and the controller reaches the optimal value. Save the trained DDPG controller and verify the control performance through testing. Set the breathing rate and tidal volume to 20 cycles / min and 1 L / cycle respectively. The simulation results are as follows: Figure 6 shown.

[0110] like Figure 6 As shown in the figure, during the breathing process, the DDPG controller can provide oxygen in time according to the pressure difference state. Under the initial conditions, it can quickly control the pressure difference within the ideal range without adjustment, and the maximum negative pressure of the breathing resistance value is only about 20 Pa, which proves the great advantage of the intelligent controller.

[0111] In order to further verify the regulation effect of the deep reinforcement learning controller, this application investigates the changes in human respiratory rate and tidal volume in different scenarios, and applies the controller in actual scenarios through simulation. The simulation results are as follows: Figure 7-Figure 10 shown.

[0112] Figure 7 A long-distance running scenario was simulated. To increase the contrast, the first 15 seconds showed the pressure regulation effect of the human body under normal breathing conditions, and the last 15 seconds showed the breathing conditions under long-distance running. At this time, the breathing frequency and tidal volume increased at the same time, and the breathing resistance also increased accordingly. Figure 8 A sleeping scene was simulated, during which both the respiratory rate and tidal volume decreased simultaneously, as shown in the last 15 seconds of the figure, and the respiratory resistance decreased year-on-year. Fig. 9A diving scenario was simulated. At this time, the breathing rate decreased and the tidal volume increased. The first 10 seconds were normal. The increase in tidal volume in 10-20 seconds was faster than the decrease in breathing rate. In the last 10 seconds, the changes in tidal volume and breathing rate were the same. Therefore, the breathing resistance in the second stage of the figure was greater than that in the first stage, while the third stage was the same as the first stage. Fig.10 The breathing state in a plateau scenario was simulated, with the respiratory rate increasing and the tidal volume decreasing. The simulation was divided into three stages, in which the increase in respiratory rate from 10 to 20 seconds was lower than the tidal volume, and the respiratory resistance was reduced. From 20 to 30 seconds, the increase in respiratory rate was the same, and the respiratory resistance was equivalent to normal.

[0113] From the above, it can be seen that the controller based on deep reinforcement intelligent learning algorithm can meet the pressure regulation requirements of the breathing chamber in a variety of scenarios, and the regulation effect has been greatly improved. No matter how the breathing frequency and tidal volume change, the inhalation resistance is only about tens of Pa, which is nearly ten times higher than the hundreds of Pa that can be achieved by the control effect based on neural network compensation. At the same time, it can also close the valve in time during the exhalation stage to stop providing oxygen.

[0114] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0115] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for controlling oxygen supply of an oxygen mask based on deep reinforcement learning, characterized in that: A neural network is used to generate a control instruction a according to an output state s of an oxygen mask oxygen supply device, and the oxygen mask control device is regulated according to the control instruction a; The neural network is trained through the following steps: S1, determine the output state s of the oxygen mask oxygen supply device, including: the error between the expected signal and the actual signal of the pressure difference between the inside and outside of the mask, the error derivative and the respiratory flow; S2, based on the output state s of the oxygen mask oxygen supply device, select action a according to the 1-ε greedy strategy, execute action a to obtain the corresponding reward r and the state s' at the next moment, and obtain the four-tuple (s, a, r, s'); S3. Randomly extract quadruple data and use the deep reinforcement learning algorithm to train the neural network until the iteration requirements are met.

2. The oxygen mask oxygen supply control method according to claim 1, characterized in that: The oxygen mask oxygen supply device comprises a proportional solenoid valve and a breathing structure, and the proportional solenoid valve realizes the oxygen supply regulation of the oxygen mask control device according to the control instruction a.

3. The oxygen mask oxygen supply control method according to claim 2, characterized in that: The breathing structure includes a simulated lung, a breathing cavity and an exhalation valve. The error and error derivative between the expected signal and the actual signal of the pressure difference between the inside and outside of the mask are obtained through the breathing cavity, and the breathing flow is obtained according to the simulated lung; the expression is: In the formula, s1(k) represents the first state at time k, s2(k) represents the second state at time k, s3(k) represents the third state at time k, and y r represents the expected signal of the pressure difference between the inside and outside of the mask, y(k) represents the actual signal of the pressure difference between the inside and outside of the mask at time k, y(k)-y(k-1) represents the error of the actual signal of the pressure difference between the inside and outside of the mask at time k minus the actual signal at the previous moment, which reflects the change trend of the error, Q b (k) represents respiratory flow.

4. The oxygen mask oxygen supply control method according to claim 1, characterized in that: The expression of 1-ε greedy strategy is: Where ε is the exploration parameter and A(s) represents the action set.

5. The oxygen mask oxygen supply control method according to claim 1, characterized in that: When executing action a, the corresponding reward is obtained according to the following reward function: Wherein, s1 represents the first state, s2 represents the second state, u represents the evaluation of energy consumption, and p and q represent weight coefficients.

6. The oxygen mask oxygen supply control method according to claim 1, characterized in that: When executing action a, the corresponding reward is obtained according to the following reward function: r(k)=-|s1|-m|s2| In the formula, s1 represents the first state, s2 represents the second state, and m represents the adjustment coefficient of the second state.

7. The oxygen mask oxygen supply control method according to claim 1, characterized in that: In each loop iteration, the termination condition is, Achieve the goal: the pressure difference between the inside and outside of the mask is controlled below 0.3KPa for two consecutive breathing cycles; Stop loss in time: 500 consecutive pressure difference data are greater than 10KPa; Or time limit: the training duration of a cycle has reached 30s and the termination conditions of reaching the goal or stopping loss in time are not met.

8. The oxygen mask oxygen supply control method according to claim 1, characterized in that: The four-tuple includes simulation data and real-time control data.

9. An oxygen mask oxygen supply control system based on deep reinforcement learning, characterized in that: It includes an oxygen supply controller and an oxygen mask, wherein the output end of the oxygen supply controller is connected to the control end of the oxygen mask, and the output end of the oxygen mask is connected to the input end of the oxygen supply controller; The oxygen supply controller internally stores the deep reinforcement learning algorithm in the oxygen mask oxygen supply control method based on deep reinforcement learning described in any one of claims 1-8; it is used to generate a control instruction a according to the output state s of the oxygen mask oxygen supply device using the deep reinforcement learning algorithm, and regulate the control end of the oxygen mask according to the control instruction a.

10. The oxygen mask oxygen supply control system according to claim 9, characterized in that: The oxygen mask oxygen supply device comprises a proportional solenoid valve and a breathing structure, and the breathing structure comprises a simulated lung, a breathing cavity and an exhalation valve.

Images