Diesel injector injection control method based on DDPG reinforcement learning

Abstract

The invention belongs to the technical field of diesel engines, and particularly relates to a diesel injector injection control method based on DDPG reinforcement learning, comprising the following steps: S1, constructing a diesel injector injection control environment model; s2, establishing a DDPG reinforcement learning algorithm model; s3, performing offline training to obtain an optimal injection control strategy model; and S4, performing online application and fine adjustment on the model in actual operation. Optimal regulation and control of the diesel engine oil injection strategy can be achieved under the complex and changeable working conditions, the influence of the operation working conditions on oil injection parameters is fully considered, and therefore the fuel economy and the emission performance of the diesel engine are effectively improved, and good engineering application value is achieved.

Description

Technical Field

[0001] This invention belongs to the field of diesel engine technology, specifically relating to a diesel engine injector injection control method based on DDPG reinforcement learning. Background Technology

[0002] With the transformation of the global energy structure and the increasing demands for environmental protection, diesel engines, as a key representative of internal combustion engines, are widely used in transportation, industry, and agriculture. However, emissions and fuel consumption issues have become major bottlenecks for the sustainable development of diesel engines. To address environmental pollution and energy efficiency, the optimization of diesel engine injection technology has become a current research hotspot. Injection control directly affects not only the combustion process, emissions performance, and economy of diesel engines, but also their power output and long-term stability. However, traditional injection control methods mostly rely on fixed models or empirical settings, making it difficult to achieve real-time and precise control under complex operating conditions. Especially in rapidly changing working environments, how to automatically adjust the injection strategy according to different operating conditions remains a significant challenge for current injection control technology.

[0003] To address the aforementioned technical problems, this invention proposes a diesel engine injector injection control method based on DDPG reinforcement learning. This method combines the characteristics of reinforcement learning with diesel engine injection control, and utilizes the DDPG algorithm to dynamically adjust and optimize the injection process, thereby achieving precise control of diesel engine injector injection under various operating conditions.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a diesel engine injector injection control method based on DDPG reinforcement learning, so as to effectively improve the accuracy of injection control and meet the high requirements of diesel engine injection performance under complex working conditions.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A diesel engine injector injection control method based on DDPG reinforcement learning includes the following steps: S1. Construct a diesel engine injector injection control environment model; S2. Construct the DDPG reinforcement learning algorithm model; S3, offline training; S4, Online Applications and Fine-tuning.

[0007] As a preferred option, the diesel engine injector injection control environment model is constructed in S1, which specifically includes the following steps: S101. Define the state space S; The state space S includes: cylinder pressure p e Required torque M, diesel engine cooling water temperature T W Exhaust temperature T O Effective fuel consumption rate; S102, Define action space A; Action space A includes: pre-injection pressure p pre Commands, pre-injection quantity command, pre-injection timing command, main injection pressure p main Injection commands, main injection quantity commands, and main injection timing commands; S103. Construct the penalty function r at any time t. t Used to guide the learning direction of the DDPG reinforcement learning algorithm; penalty function r t The formula is as follows: r t = -[ w1 * ABS(p emax -p e ) + w2 * ABS(T Wmax -T W ) + w3 * ABS(T Omax -T O ) + w4* (BSFC) + w5 * ABS(MM O )]; Where, p emax T Wmax T Omax M O These are the upper limit of cylinder pressure to prevent knocking, the maximum safe temperature of diesel engine coolant, the maximum safe temperature of exhaust, and the output torque of diesel engine, respectively; w1, w2, w3, w4, and w5 are weighting coefficients used to adjust the priority of different targets.

[0008] As a preferred option, the DDPG reinforcement learning algorithm model is constructed in S2, which specifically includes the following steps: S201. Construct an online action network μ, input S into μ, and obtain the output: A = μ(S); S202. Construct the target action network μ', which has the same structure as μ. Input S into μ' and obtain the output: A'=μ'(S); S203. Construct an online evaluation network Θ, input S and A into Θ, and output the Q value of the online evaluation network: q1=Θ(S,A); S204. Construct the target evaluation network Θ'. Θ' has the same structure as Θ. Input S and A into Θ' and output the Q value of the target evaluation network. q2=Θ'(S,A); S205. All network weights are initialized randomly or through pre-training.

[0009] Preferably, offline training is performed in S3, which includes the following steps: S301. Define the state space S at any time t. t and action space A t ; S302, Online Action Network μ based on the current state S t Select action A t And add exploration noise; S303, Diesel engine action A t That is, control the diesel engine to run according to this action for one time step, and collect the state S at time t+1. t+1 and time t reward r t ; S304, Transfer data (S t A t , r t , S t+1 Stored in the experience replay buffer; S305. Randomly sample small batches of data N×(S) from the experience playback buffer. t A t , r t , S t+1 ); S306. Input the data sampled in S305, calculate the TD target, update the online evaluation network Θ by minimizing the mean square errors of q1 and q2, and update the online action network μ by using the policy gradient ascent method. S307, soft update the target action network μ' and the target evaluation network Θ' to maintain the stability of the learning process; S308. Repeat steps S301-S307 until the policy performance converges, i.e., the penalty function r... t The mean value has remained at a low level.

[0010] As a preferred option, online application and fine-tuning are performed in S4, specifically including the following steps: S401. Deploy the trained μ' to the vehicle ECU device; S402, ECU real-time acquisition of engine status at time t. t ; S403, S t The optimal action is obtained by inputting the already deployed μ', without adding exploration noise; S404: Convert the optimal action into a specific control signal to drive the diesel engine injector to execute; S405. In actual operation, continue to fine-tune online at a low learning rate to adapt to the personalized characteristics of specific vehicles and drivers.

[0011] Compared with the prior art, the present invention has the following beneficial effects: (1) The diesel engine injector injection control method based on DDPG reinforcement learning of the present invention, by combining the DDPG reinforcement learning algorithm, continuously interacts with the environment to adjust the control strategy, and can accurately control the injection of diesel engine injectors under various complex working conditions.

[0012] (2) The diesel engine injector injection control method based on DDPG reinforcement learning of the present invention can adjust the injection pressure, injection quantity and injection timing according to the real-time operating conditions of the diesel engine in the application of the injector, thereby ensuring the best injection effect under the operating conditions.

[0013] (3) The diesel engine injector injection control method based on DDPG reinforcement learning of the present invention fully leverages the advantages of DDPG reinforcement learning in diesel engine injector injection control, improves the adaptability and accuracy of diesel engine injection strategy, and ensures the efficient and stable operation of diesel engine under complex and variable working conditions, thereby achieving dual optimization of diesel engine performance and emissions. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the DDPG reinforcement learning algorithm of the present invention. Detailed Implementation

[0015] The technical solution of this invention patent will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0016] In the description of this invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention.

[0017] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0018] See attached document Figure 1 A diesel engine injector injection control method based on DDPG reinforcement learning includes the following steps: S1. Construct a diesel engine injector injection control environment model: S101. Define the state space S, which contains parameters that characterize the instantaneous operating conditions and performance of the diesel engine; S102. Define the motion space A, which contains the continuous control quantities that the controller needs to output; S103. Construct the penalty function r at any time t. t This is used to guide the learning direction of the DDPG reinforcement learning algorithm. S2. Construct the DDPG reinforcement learning algorithm model: S201. Construct an online action network μ, input S into μ, and obtain the output: A = μ(S); S202. Construct the target action network μ', which has the same structure as μ. Input S into μ' and obtain the output: A'=μ'(S); S203. Construct an online evaluation network Θ, input S and A into Θ, and output the Q value of the online evaluation network: q1=Θ(S,A); S204. Construct the target evaluation network Θ'. Θ' has the same structure as Θ. Input S and A into Θ' and output the Q value of the target evaluation network. q2=Θ'(S,A); S205. All network weights are initialized randomly or through pre-training. S3, Offline Training: S301. Define the state space S at any time t. t and action space A t ; S302, Online Action Network μ based on the current state S t Select action A t And add exploration noise; S303, Diesel engine action A tThat is, control the diesel engine to run according to this action for one time step, and collect the state S at time t+1. t+1 and time t reward r t ; S304, Transfer data (S t A t , r t , S t+1 Stored in the experience replay buffer; S305. Randomly sample small batches of data N×(S) from the experience playback buffer. t A t , r t , S t+1 ); S306. Input the data sampled in S305, calculate the TD target, update the online evaluation network Θ by minimizing the mean square errors of q1 and q2, and update the online action network μ by using the policy gradient ascent method. S307, soft update the target action network μ' and the target evaluation network Θ' to maintain the stability of the learning process; S308. Repeat steps S301-S307 until the policy performance converges, i.e., the penalty function r... t The mean value remained stable at a low level; S4. Online Applications and Fine-tuning: S401. Deploy the trained μ' to the vehicle ECU device; S402, ECU real-time acquisition of engine status at time t. t ; S403, S t The optimal action is obtained by inputting the already deployed μ', without adding exploration noise; S404: Convert the optimal action into a specific control signal to drive the diesel engine injector to execute; S405. In actual operation, continue to fine-tune online at a low learning rate to adapt to the personalized characteristics of specific vehicles and drivers.

[0019] This embodiment demonstrates the DDPG reinforcement learning algorithm, see attached diagram. Figure 2 .

[0020] S101 defines a state space S, which includes, but is not limited to: cylinder pressure p. e Required torque M, diesel engine cooling water temperature T W Exhaust temperature T O 1. Effective fuel consumption rate (BSFC).

[0021] S102 defines the action space A, which contains the continuous control quantities that the controller needs to output, specifically including: pre-injection pressure ppre Commands, pre-injection quantity command, pre-injection timing command, main injection pressure p main Commands, main injection quantity commands, and main injection timing commands; action commands are restricted to the physical range of safe operation of the diesel engine before passing through the controller.

[0022] Construct the penalty function r at any time t in S103. t Used to guide the learning direction of the DDPG reinforcement learning algorithm; penalty function r t The formula is as follows: r t = -[ w1 * ABS(p emax -p e ) + w2 * ABS(T Wmax -T W ) + w3 * ABS(T Omax -T O ) + w4* (BSFC) + w5 * ABS(MM O )]; Where, p emax T Wmax T Omax M O These are the upper limit of cylinder pressure to prevent knocking, the maximum safe temperature of diesel engine coolant, the maximum safe temperature of exhaust, and the output torque of diesel engine, respectively; w1, w2, w3, w4, and w5 are weighting coefficients used to adjust the priority of different targets.

[0023] Traditional methods typically rely on experience or fixed models, making it difficult to handle varying operating conditions and complex nonlinear characteristics. The DDPG algorithm, however, possesses adaptive learning and optimization capabilities. It can continuously adjust its control strategy through interaction with the environment, thereby learning the optimal fuel injection control strategy even without a defined model. Particularly in injector applications, DDPG can adjust injection pressure, injection quantity, and injection timing based on the real-time operating conditions of the diesel engine, ensuring optimal fuel injection performance under all operating conditions.

[0024] This invention, by combining the DDPG reinforcement learning algorithm, can precisely control the injection of diesel engine injectors under various complex operating conditions. It fully leverages the advantages of the DDPG reinforcement learning method in diesel engine injector injection control, improves the adaptability and accuracy of the diesel engine injection strategy, and ensures the efficient and stable operation of the diesel engine under complex and variable operating conditions. This achieves dual optimization of diesel engine performance and emissions, resulting in significant economic and environmental benefits.

[0025] Those skilled in the art will recognize that the technical solutions described in the above embodiments can be executed by a computer program driving the corresponding hardware device to perform all or part of the steps, and the relevant program can be stored in a non-transitory computer-readable storage medium, including but not limited to disks, optical storage media, read-only memory or random access memory.

[0026] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A diesel engine injector injection control method based on DDPG reinforcement learning, characterized by, The method comprises the following steps: S1, constructing a diesel engine injector injection control environment model; S2, constructing a DDPG reinforcement learning algorithm model; S3, offline training; S4, online application and fine-tuning.

2. The DDPG reinforcement learning-based diesel fuel injector injection control method according to claim 1, characterized by, The diesel engine injector injection control environment model in S1 comprises the following steps: S101, defining a state space S; The state space S includes: cylinder pressure p e , required torque M, diesel engine cooling water temperature T W , exhaust gas temperature T O , effective fuel consumption rate; S102, defining an action space A; The action space A includes: pre-injection pressure p pre command, pre-injection fuel quantity command, pre-injection timing command, main injection pressure p main command, main injection fuel quantity command, and main injection fuel timing command; S103、constructing a penalty function r at any t moment t , for guiding the learning direction of DDPG reinforcement learning algorithm; penalty function r t The formula is as follows: r t = -[ w1 * ABS(p emax -p e ) + w2 * ABS(T Wmax -T W ) + w3 * ABS(T Omax -T O ) + w4 *(BSFC) + w5 * ABS(M-M O )]; wherein p emax , T Wmax , T Omax , M O are the upper limit of the cylinder pressure for preventing knock, the maximum safe temperature of the diesel engine cooling water, the maximum safe temperature of the exhaust gas, and the output torque of the diesel engine, respectively; w1, w2, w3, w4, and w5 are weight coefficients for adjusting the priority of different targets.

3. The DDPG reinforcement learning-based diesel fuel injector injection control method according to claim 2, characterized by, The DDPG reinforcement learning algorithm model in S2 comprises the following steps: S201, constructing an online action network μ, inputting S into μ to obtain an output: A=μ(S); S202, constructing a target action network μ', which has the same structure as μ, inputting S into μ' to obtain an output: A'=μ'(S); S203, constructing an online evaluation network Θ, inputting S and A into Θ to output the Q value of the online evaluation network: q1=Θ(S,A); S204, constructing a target evaluation network Θ', which has the same structure as Θ, inputting S and A into Θ' to output the Q value of the target evaluation network: q2=Θ'(S,A); S205, all network weights are randomly initialized or initialized through pre-training.

4. The DDPG reinforcement learning-based diesel fuel injector injection control method according to claim 3, characterized by, The offline training in S3 comprises the following steps: S301、define the state space S at any time t t and action space A t ; S302, the online action network μ updates its current state S t selecting an action A t and adding exploration noise; S303, the diesel engine performs action A t , i.e. controls the diesel engine to run for one time step according to the action, collects the state S at t+1 t+1 and the reward r at t t ; S304, store the data (S t , A t , r t , S t+1 ) into the experience replay buffer; S305, randomly sample mini-batch data N x (S t , A t , r t , S t+1 ) from experience replay buffer; S306, inputting the data sampled in S305, calculating the TD target, updating the online evaluation network Θ by minimizing the mean square error of q1 and q2, and updating the online action network μ by the policy gradient ascent method; S307, soft updating the target action network μ' and the target evaluation network Θ' to maintain the stability of the learning process; S308, repeat steps S301-S307 until the policy performance converges, i.e. the mean of the penalty function r t stabilizes at a lower level.

5. The DDPG reinforcement learning-based diesel fuel injector injection control method according to claim 4, characterized by, The online application and fine-tuning in S4 comprise the following steps: S401, deploying the trained μ' to the vehicle-mounted ECU device; S402, the ECU collects the engine state at time t in real time S t ; S403, S t The optimal action is obtained by inputting the already deployed μ', without adding exploration noise; S404, converting the optimal action into a specific control signal to drive the diesel engine injector to execute; S405, in actual operation, continue to fine-tune online with a lower learning rate to adapt to the individual characteristics of specific vehicles and drivers.

Images