Model training method, control method, device, medium and program product

Abstract

The embodiment of the invention discloses a model training method, a control method, equipment, a medium and a program product. The method comprises the steps of obtaining a training sample set; inputting a next training state in the training sample into the initial action network to obtain a next training action output by the initial action network; inputting the next training action, the current training state in the training sample, the current training action and the next training state into the initial value network to obtain a current target Q value output by the initial value network; if the iteration termination condition is not met, updating the initial value network, obtaining an updated initial value network, updating the initial action network and obtaining an updated initial action network according to the current target Q value, and returning to execute the step of obtaining the training sample set; and if the iteration termination condition is satisfied, determining the initial action network as a target control model. According to the model training method, rapid convergence of the target control model is realized, and the stability of the controlled equipment is improved.

Description

Technical Field

[0001] The present invention relates to the field of industrial control technology, and in particular to a model training method, a control method, a device, a medium and a program product. Background Art

[0002] With the development of big data technology, industrial control can be achieved through neural networks.

[0003] In related technologies, neural networks can be integrated into industrial control systems. The control process is as follows: the neural network can output control parameters to control the output parameters of the controlled devices in the industrial control system. If the output parameters of the controlled devices do not meet the conditions, the neural network can be updated to obtain the updated neural network, and the process returns to the first step until the output parameters of the controlled devices meet the conditions, at which point the neural network updates cease.

[0004] However, during the above online training process, since the training samples are data actually sampled from the industrial control system and are limited in number, the neural network may converge slowly. In addition, the neural network may have unstable outputs during the training process, resulting in poor stability of the controlled equipment. Summary of the Invention

[0005] The present invention provides a model training method, a control method, a device, a medium and a program product to solve the technical problems of slow neural network convergence speed and poor stability of controlled devices caused by the model training method in the related art.

[0006] According to one aspect of the present invention, a model training method is provided, the method comprising:

[0007] Obtaining a training sample set; wherein the training sample set includes a first type of training samples and a second type of training samples, the first type of training samples are data generated by a sample generation model, and the second type of training samples are data generated after the controlled device is actually operated;

[0008] Inputting the next training state in the training sample into the initial action network to obtain the next training action output by the initial action network;

[0009] Inputting the next training action, the current training state in the training sample, the current training action and the next training state into an initial value network to obtain a current target Q value output by the initial value network;

[0010] If the iteration termination condition is not met, then according to the current target Q value, the initial value network is updated to obtain an updated initial value network, the initial action network is updated to obtain an updated initial action network, and the process returns to the step of "obtaining a training sample set";

[0011] If the iteration termination condition is met, the initial action network is determined as a target control model; wherein the target control model is used to determine a current action input to the controlled device according to the current state of the controlled device.

[0012] According to another aspect of the present invention, there is provided a control method, the method comprising:

[0013] Obtaining the current status of the controlled device;

[0014] Inputting the current state into a target control model to obtain a current action output by the target control model; wherein the target control model is a target control model obtained according to the model training method described in the above embodiment;

[0015] The current action is input into the controlled device.

[0016] According to another aspect of the present invention, a model training device is provided, the device comprising:

[0017] A first acquisition module is configured to acquire a training sample set; wherein the training sample set includes a first type of training samples and a second type of training samples, wherein the first type of training samples are data generated by a sample generation model, and the second type of training samples are data generated after the controlled device is actually operated;

[0018] A first determination module is configured to input the next training state in the training sample into the initial action network to obtain the next training action output by the initial action network;

[0019] A second determination module is configured to input the next training action, the current training state in the training sample, the current training action, and the next training state into an initial value network to obtain a current target Q value output by the initial value network;

[0020] An updating module is configured to update the initial value network according to the current target Q value to obtain an updated initial value network, update the initial action network to obtain an updated initial action network, and return to the step of "obtaining a training sample set" if the iteration termination condition is not met;

[0021] The third determining module is configured to determine the initial action network as a target control model if an iteration termination condition is met; wherein the target control model is configured to determine a current action input to the controlled device according to a current state of the controlled device.

[0022] According to another aspect of the present invention, there is provided a control device, the device comprising:

[0023] The second acquisition module is used to obtain the current status of the controlled device;

[0024] a first input module, configured to input the current state into a target control model to obtain a current action output by the target control model; wherein the target control model is a target control model obtained according to the model training method in any of the above embodiments;

[0025] The second input module is configured to input the current action into the controlled device.

[0026] According to another aspect of the present invention, an electronic device is provided, comprising:

[0027] at least one processor; and

[0028] a memory communicatively connected to the at least one processor; wherein,

[0029] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the model training method or control method described in any embodiment of the present invention.

[0030] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a processor to implement the model training method or control method described in any embodiment of the present invention when executed.

[0031] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the model training method or control method described in any embodiment of the present invention.

[0032] The technical solution of the embodiment of the present invention is that, on the one hand, the training sample set includes a first type of training samples generated by the sample generation model and a second type of training samples generated after the controlled device is actually operated, which reduces the dependence of the model training process on the real environment. At the same time, the sample generation model can try high-risk actions in a virtual environment to obtain corresponding first type of training samples, thereby avoiding damage to the controlled device, expanding the training sample set, improving data utilization, and combining the diversity of the first type of training samples and the reliability and accuracy of the second type of training samples in the model training process, thereby achieving rapid convergence of the target control model; on the other hand, the method uses an offline training method to obtain the target control model, thereby avoiding the unstable output of the control model in the online training method and improving the stability of the controlled device; on the other hand, the method uses the action network-value network model for model training, thereby improving the accuracy of the trained target control model, thereby improving the accuracy of controlling the controlled device using the target control model.

[0033] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 This is a flow chart of a model training method provided by an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of a control system provided by an embodiment of the present invention;

[0037] Figure 3 is a schematic diagram of another control system provided by an embodiment of the present invention;

[0038] Figure 4 is a flowchart of another model training method provided by an embodiment of the present invention;

[0039] Figure 5 is a schematic diagram of each model provided by an embodiment of the present invention;

[0040] Figure 6 is a flow chart of a control method provided by an embodiment of the present invention;

[0041] Figure 7It is a schematic diagram of the response curve of the control system;

[0042] Figure 8 Schematic diagram of a model training device provided by an embodiment of the present invention;

[0043] Figure 9 is a structural diagram of a control device provided by an embodiment of the present invention;

[0044] Figure 10 It is a structural diagram of an electronic device for implementing the control method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0046] It should be noted that the terms "initial", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the embodiments of the present invention comply with the relevant provisions of national laws and regulations.

[0047] The model training method provided in this embodiment is applicable to scenarios where a model for controlling a controlled device is trained. The controlled device in this embodiment can be any device in the industrial control field that can be controlled by a network model, or a system composed of various components. Alternatively, the controlled device in this embodiment can be a controller. For another example, the controlled device in this embodiment can include a controller and an execution device.

[0048] Optionally, the controlled device in this embodiment includes: a proportional integral (PI) controller, a power source for outputting a radio frequency signal, and a radio frequency signal receiving device connected in sequence. That is, the controller in this embodiment may be a PI controller, and the execution device may include a power source for outputting a radio frequency signal and a radio frequency signal receiving device.

[0049] The mathematical description of the PI controller is: out =K p ×error+K i ×∑error, where K p represents the scale parameter, K i represents the integral parameter, error represents the error, Y out represents the output of the PI controller, and ∑error represents the integral of the error.

[0050] Furthermore, the radio frequency signal receiving device in this embodiment can be an accelerator. An accelerator is a device that uses electromagnetic fields to allow charged particles to obtain high energy and be accelerated to a high-speed state close to the speed of light, and is used to accelerate, manipulate and study particles. In the accelerator structure, the radio frequency electric field is designed with a periodically changing electric field direction and intensity gradient, so that the charged particles obtain continuous energy gain when crossing the acceleration gap, and eventually form a high-energy, high-current particle beam. This technology has extended from basic scientific research to industrial applications in multiple fields: Basic scientific research field: high-energy accelerators provide key experimental means for nuclear physics and particle physics research; industrial material modification field: the ion beams generated by accelerators show special value in industrial production.

[0051] The Low-Level Radio Frequency (LLRF) control system is a core component for achieving precise control of the RF power signal amplitude and phase, directly impacting the stability of the accelerating electric field and the quality of the particle beam. The LLRF control system includes a PI controller.

[0052] In related technologies, PI controllers achieve feedback regulation through fixed parameters, but they have the following limitations under large disturbances or complex working conditions: 1. Insufficient anti-disturbance capability: When the resonant frequency of the accelerator cavity drifts due to factors such as thermal deformation and load fluctuations, the fixed-parameter PI controller has difficulty quickly tracking dynamic changes, which can easily cause amplitude overshoot or phase loss. 2. Limited control accuracy: Analog low-level systems rely on discrete devices to adjust voltage, which has problems with temperature drift and noise interference. Although closed-loop control based on field-programmable gate arrays (FPGAs) in digital solutions improves stability, it has poor adaptability to nonlinear responses. 3. Rigid parameter optimization: Existing digital low-level systems require pre-setting PI parameters and cannot be dynamically adjusted according to real-time working conditions, resulting in reduced control efficiency of the accelerator in scenarios such as variable energy and beam modulation. To address the above problems, the accelerator can be controlled by online training of neural networks. However, during the online training process, since the training samples are data actually sampled from the industrial control system and are limited in number, the neural network may converge slowly. In addition, the neural network may have unstable outputs during the training process, resulting in poor stability of the controlled equipment.

[0053] Based on this, this embodiment provides a model training method to solve the above problems.

[0054] Figure 1 This is a flow chart of a model training method provided by an embodiment of the present invention. This embodiment is applicable to scenarios where a model for controlling a controlled device is trained. The method can be executed by a model training device, which can be implemented in the form of hardware and / or software. The model training device can be configured in an electronic device, such as a computer device. Figure 1 As shown, the method includes the following steps 101 to 105.

[0055] Step 101: Obtain a training sample set.

[0056] The training sample set includes a first type of training samples and a second type of training samples. The first type of training samples are data generated by a sample generation model, and the second type of training samples are data generated after the controlled device is actually operated.

[0057] In order to improve the convergence speed of the control model during training, the training sample set in this embodiment consists of two types of training samples: a first type of training samples generated by the sample generation model, and a second type of training samples generated after the controlled device is actually running.

[0058] Optionally, the training sample in this embodiment includes: (current training state, current training action, reward value, next training state). In a scenario where the controlled device includes a PI controller, a power source, and an accelerator, the current training state includes: a current training error and an integral of the current training error. The current training error represents the difference between the training set value and the training output, and the integral of the current training error represents the cumulative sum of historical training errors. The current training action includes: a current training ratio parameter and a current training integral parameter.

[0059] The sample generation model in this embodiment is a pre-trained model. The training process of the sample generation model can be: using the dynamic model of the controlled device and real-time monitoring data to establish a dynamic model of the state space that includes disturbance factors such as thermal deformation and load fluctuation, and through online rolling optimization to predict the system response and generate control action boundary constraints. In a scenario where the controlled device includes a PI controller, a power source, and an accelerator, when training the sample generation model, the dynamic model of the electromagnetic field of the accelerator cavity (such as a transfer function) and real-time monitoring data is used to establish the dynamic model.

[0060] During the training of the sample generation model, a random strategy is used to collect initial transition samples (current state, current action, reward, next state) in a real environment. Subsequently, the sample generation model is trained through supervised learning to accurately predict: state transitions: Δstate(next state) = f(state, action); immediate reward: reward = r(state, action). In other words, the function of the sample generation model in this embodiment is to predict the next training state and immediate reward after performing an action (current training action) in a given training state (current training state).

[0061] For example, in a scenario where the controlled device includes a PI controller, a power source, and an accelerator, the reward value in this embodiment can be determined by the following formula: 2 +β*u 2 ), where error represents the error and u represents the RF output of the power source. The reward value consists of two parts:

[0062] 1) Error square term error 2 : represents the square of the deviation between the expected parameter (Setpoint) and the output parameter. The larger the error, the stronger the penalty. The value of λ can be 0.9.

[0063] 2) Control quantity squared term: Penalizes the magnitude of the control quantity (u) to avoid excessive control action. β can be set to 0.01.

[0064] Optionally, the sample generation model in this embodiment may be a neural network model with two hidden layers and a ReLU activation function. The learning rate during the sample generation model training process may be 1e-3.

[0065] After training the sample generation model, the first type of training samples can be generated using the sample generation model. The inputs to the sample generation model include the current training state and the current training action; the outputs include the next training state and the reward value. The process of generating the first type of training samples can include the following: 1. Initial state: Each virtual trajectory begins with the real environment reset state; 2. Generating the current training action and the current training state; 3. Predicting the next training state and the reward value; 4. Termination condition: The virtual trajectory is fixed to a preset value, for example, 20 steps.

[0066] The generation process of the first type of training samples has the following advantages: 1. It expands the training sample set and reduces the interaction with the real environment during the training sample generation process; 2. It is suitable for high-cost real control scenarios, and high-risk actions can be attempted in the virtual environment to avoid damage to the real system, thereby increasing the diversity of the first type of training samples; 3. It improves data utilization; 4. Error propagation is limited: the virtual trajectory only runs for 20 steps to avoid state drift caused by accumulation of model errors.

[0067] The second type of training samples in this embodiment are empirical data collected from real-world interactions, that is, data generated after the controlled device actually operates. It can be understood that the second type of training samples are reliable and accurate.

[0068] Optionally, the first type of training samples in this embodiment may be stored in a virtual buffer (VirtualBuffer) of the electronic device. The second type of training samples in this embodiment may be stored in a real buffer (RealBuffer) of the electronic device.

[0069] Therefore, the training sample set in this embodiment combines the reliability and accuracy of real data with the efficiency and diversity of virtual data, achieving rapid convergence and stable training of the control model in complex control tasks, significantly enhancing the robustness of the control model, and not only breaking through the traditional control theory's reliance on precise mathematical models, but also realizing subsequent real-time optimal control of the controlled equipment.

[0070] Furthermore, the training sample in this embodiment may further include a terminal state flag done. If done is false, it indicates that it is not a terminal state, and if done is true, it indicates that it is a terminal state.

[0071] Optionally, the ratio of the first category of training samples to the second category of training samples in the training sample set can be 1:1. That is, assuming the number of training sample sets is batch_size, batch_size / 2 of the first category of training samples are obtained from the virtual buffer, and batch_size / 2 of the second category of training samples are obtained from the real buffer. For example, batch_size in this embodiment can be 128. This implementation method can balance the accuracy of the training sample set and the exploration efficiency.

[0072] Step 102: Input the next training state in the training sample into the initial action network to obtain the next training action output by the initial action network.

[0073] In this embodiment, a reinforcement learning framework is used for model training, for example, the Deep Deterministic Policy Gradient (DDPG) framework. More specifically, the model training is performed using an action network-value network model. In step 102, the next training state from the first training sample and the second training sample is input into the initial action network, and the next training action output by the initial action network is obtained.

[0074] The role of the action network is to output a deterministic action based on the input state. Specifically, the input of the initial action network includes the next training state, and the output includes the next training action. The next training state includes the next training error and the integral of the next training error. The next training action includes the next training scale parameter and the next training integral parameter.

[0075] Optionally, the initial action network in this embodiment can use a Tanh activation function to constrain its output to the range [-1, 1], and then map it to the actual action range [0, 2] through a linear transformation. Relying on the diverse data in the virtual buffer (i.e., the first type of training samples), the initial action network in this embodiment can adopt a non-random exploration mechanism.

[0076] Step 103: Input the next training action, the current training state in the training sample, the current training action and the next training state into the initial value network to obtain the current target Q value output by the initial value network.

[0077] The value network in this embodiment outputs the expected cumulative reward for a state-action pair, i.e., the target Q-value. The inputs to the initial value network are the current training state, the current training action, the next training state, and the next training action, and the output is the current target Q-value. The initial value network in this embodiment can directly evaluate the value of deterministic actions without randomness.

[0078] Step 104: If the iteration termination condition is not met, the initial value network is updated according to the current target Q value to obtain an updated initial value network, the initial action network is updated to obtain an updated initial action network, and the process returns to step 101.

[0079] The iteration termination condition in this embodiment may be a preset number of iterations, a preset iteration duration, the current target Q value no longer increasing, or the current target Q value being greater than a preset value.

[0080] If the iteration termination condition is not met, the initial value network is updated to obtain an updated initial value network, the initial action network is updated to obtain an updated initial action network, and the process returns to step 101 to continue acquiring training sample sets. It is understood that in one iteration, the initial action network is the updated initial action network from the previous iteration, and the initial value network is the updated initial value network from the previous iteration.

[0081] Through steps 102 to 104 above, the initial value network gradually learns the true Q value of the state-action pair, provides an accurate gradient direction for the strategy optimization of the action network, and ultimately achieves efficient control of the system.

[0082] Step 105: If the iteration termination condition is met, the initial action network is determined as the target control model.

[0083] The target control model is used to determine the current action of the input controlled device according to the current state of the controlled device.

[0084] When the iteration termination conditions are met, the initial action network is determined as the target control model. Subsequently, the controlled device can be controlled based on the target control model. The input of the target control model is the current state of the controlled device, and the output is the current action of the controlled device.

[0085] In steps 102 to 105, an action network-value network architecture is adopted. Through the synergistic effect of these two networks: the action network obtains the next training action, and the value network evaluates its Q value, forming a closed loop of policy optimization, and then training the target control model to achieve adaptive adjustment of the input parameters of the controlled device.

[0086] In a scenario where the controlled device includes a PI controller, a power source, and an accelerator, the current state includes: a current error and an integral of the current error; and the current action includes: a current proportional parameter and a current integral parameter. Figure 2 Schematic diagram of a control system provided by an embodiment of the present invention. Figure 2As shown, the input of the target control model is the current error and the integral of the current error. The current error is determined according to the current desired parameters and the output parameters of the accelerator. The output of the target control model is the current integral parameter and the current proportional parameter of the input PI controller. The PI controller can input control parameters to the power source according to the current integral parameter, the current proportional parameter, the current error and the integral of the current error, and the power source feeds a radio frequency signal to the accelerator cavity under the control of the control parameters. In this embodiment, the output parameter of the accelerator is the cavity pressure signal actually coupled from the accelerator cavity. The actually coupled cavity pressure signal can also be called a sampled cavity pressure signal (pick-up cavity pressure signal).

[0087] Please continue to refer to Figure 2 In scenarios where the controlled device includes a PI controller, a power source, and an accelerator, a dual-loop control approach is employed when the target control model is subsequently used to control the controlled device. The inner loop is a PI controller that generates control parameters based on the current error, the integral of the current error, the current proportional parameter, and the current integral parameter. The outer loop uses the target control model to update the current integral and proportional parameters that are input to the PI controller. This creates an automated control-evaluation-optimization-closed-loop model.

[0088] Figure 3 FIG. 1 is a schematic diagram of another control system provided by an embodiment of the present invention. Figure 3 As shown, in a scenario where the controlled device includes a PI controller, a power source, and an accelerator (not shown), during the sample generation model training phase, the open-loop / closed-loop module is in an open-loop state and can be trained based on the collected data to obtain a sample generation model. The sample generation model generates a first type of training samples. After the controlled device in this control system is actually operating, a second type of training samples is generated.

[0089] The DDPG framework can train a target control model based on the steps from steps 102 to 105. Subsequently, when the open-loop / closed-loop module is in the closed-loop state, the controlled device can be controlled using the target control model. The specific control process is as follows. The output parameters of the sampling accelerator are converted into DC I-channel and Q-channel signals with a zero intermediate frequency after digital down-conversion. The I-channel signal is low-pass filtered (FIR) and then compared with the I-channel desired parameter (setpoint_I) to obtain the current I-channel error and the integral of the current I-channel error. The Q-channel signal is low-pass filtered (FIR) and then compared with the Q-channel desired parameter (setpoint_Q) to obtain the current Q-channel error and the integral of the current Q-channel error. The current I-channel error and the integral of the current I-channel error are input into the I-channel target control model to obtain the current I-channel proportional parameter and the current I-channel integral parameter for the I-channel PI controller. The current Q-channel error and the integral of the current Q-channel error are input into the Q-channel target control model to obtain the current Q-channel proportional parameter and the current Q-channel integral parameter for the Q-channel PI controller. The control parameters output by the I-channel PI controller are compensated by the I-channel feedforwards FF_I and FFbeam_I to obtain the I-channel control parameters. The control parameters output by the Q-channel PI controller are compensated by the Q-channel feedforwards FF_Q and FFbeam_Q to obtain the Q-channel control parameters. The I-channel and Q-channel control parameters are each passed through the output matrix and then up-converted and mixed after saturation limiting. The output control signal, after mixing to the RF frequency, passes through the RF switch and amplification chain to obtain the RF signal. The RF signal is fed into the accelerator cavity to control its operation.

[0090] based on Figure 3 In a control system, the target control model in step 105 of this embodiment can be an I-channel target control model or a Q-channel target control model. The current state may include: the current I-channel error and the integral of the current I-channel error. Correspondingly, the current action may include: the current I-channel proportional parameter and the current I-channel integral parameter. Alternatively, the current state may include: the current Q-channel error and the integral of the current Q-channel error. Correspondingly, the current action may include: the current Q-channel proportional parameter and the current Q-channel integral parameter.

[0091] Exemplarily, the proportional parameter and the integral parameter in this embodiment are both numbers greater than or equal to 0 and less than or equal to 2.

[0092] In the model training method of this embodiment, since the training sample set includes first-category training samples generated by the sample generation model, the number of real samples required for online learning is reduced. This method can improve the inefficient use of second-category training samples by using an experience replay mechanism to break down data correlations, thereby improving the stability and efficiency of model training. During training, a batch of training samples is randomly sampled from the double buffer to update the network parameters. This allows the network to learn a richer set of samples and integrate the model with the actual online environment. In scenarios where a large number of training samples are not available, this model training method demonstrates good performance stability. It has strong anti-interference capabilities and excellent robustness, achieving higher levels of robustness and steady-state accuracy under the same bandwidth and stability margin constraints. This model training method can automatically adjust the proportional-integral parameters under the condition of valid second-category training samples, enabling more complex and diverse control of the system and providing more degrees of freedom in the design and optimization of the controlled device. The method provided in this embodiment provides a design reference for adaptive stable control of amplitude and phase in low-level accelerator control systems.

[0093] The model training method provided in this embodiment, on the one hand, the training sample set includes a first type of training samples generated by the sample generation model and a second type of training samples generated after the controlled device is actually operated, which reduces the dependence of the model training process on the real environment. At the same time, the sample generation model can try high-risk actions in a virtual environment to obtain corresponding first type of training samples, thereby avoiding damage to the controlled device, expanding the training sample set, improving data utilization, and combining the diversity of the first type of training samples and the reliability and accuracy of the second type of training samples in the model training process, thereby achieving rapid convergence of the target control model; on the other hand, the method uses an offline training method to obtain the target control model, thereby avoiding the unstable output of the control model in the online training method and improving the stability of the controlled device; on the other hand, the method uses the action network-value network model for model training, thereby improving the accuracy of the trained target control model, thereby improving the accuracy of controlling the controlled device using the target control model.

[0094] Figure 4 This is a flow chart of another model training method provided by an embodiment of the present invention. Figure 1 Based on the illustrated embodiments and various optional implementations, the specific implementations of the action network and the value network are described in detail. Figure 4 As shown, the model training method provided in this embodiment includes the following steps.

[0095] Step 401: Obtain a training sample set.

[0096] The training sample set includes a first type of training samples and a second type of training samples. The first type of training samples are data generated by a sample generation model, and the second type of training samples are data generated after the controlled device is actually operated.

[0097] The implementation process and technical principle of step 402 are similar to those of step 101 and will not be repeated here.

[0098] Step 402: Input the next training states in different training samples into the initial action main network and the initial action target network respectively to obtain the next main training action output by the initial action main network and the next target training action output by the initial action target network.

[0099] In this embodiment, the initial action network includes an initial action main network and an initial action target network. The initial action network in this embodiment uses a dual network architecture of a main network and a target network to achieve soft updates of the initial action main network weights, alleviate training instability, and improve the stability of the action main network.

[0100] Step 403: Input the next training action and the next training state into the initial value target network to obtain the current first Q value output by the initial value target network.

[0101] The current first Q value is a value determined according to the next training action, the next training state, and the reward value in the training sample. The next training action includes the next main training action and the next target training action.

[0102] In this embodiment, the initial value network includes an initial value main network and an initial value target network. That is, the initial value network also adopts a dual network architecture of a main network and a target network to achieve soft updates of the initial value main network weights, alleviating the problem of training instability.

[0103] The role of the initial value target network is to calculate the expected cumulative reward Q value for the next training action and the next training state.

[0104] Optionally, when the next training action and the next training state are input into the initial value target network, the next training action and the next training state may be concatenated into a 4-dimensional vector.

[0105] The next training action in step 403 includes the next main training action and the next target training action.

[0106] Optionally, the current first Q value in this embodiment can be determined by the formula reward + (1-done) * gamma1 * target_critic (next_states, next_actions). Gamma1 is a discount factor, which indicates the importance of target_critic (next_states, next_actions). Exemplarily, the discount factor gamma1 can be 0.99. Reward represents the reward value corresponding to the next training state. If the state is a terminal state, that is, done = True, then (1-dones) is 0, and the first Q value only retains the reward value. If the state is a non-terminal state, that is, done = false, then (1-dones) is 1.

[0107] Step 404: Input the current training action and the current training state into the initial value main network to obtain the current second Q value output by the initial value main network.

[0108] The current second Q value is a value determined according to the current training action, the current training state, and the reward value in the training sample.

[0109] The role of the initial value main network is to calculate the expected cumulative reward Q value of the current training action and the current training state.

[0110] Optionally, when the current training action and the current training state are input into the initial value main network, the current training action and the current training state can be concatenated into a 4-dimensional vector.

[0111] The current second Q value can be determined by the formula reward + (1-done) * gamma2 * target_critic(states, actions). gamma2 is a discount factor that indicates the importance of target_critic(states, actions). For example, the discount factor gamma2 can be 0.99. reward represents the reward value corresponding to the current training action and the current training state. If the state is the terminal state, that is, done = True, then (1-dones) is 0, and the second Q value only retains the reward value.

[0112] During the calculation of the current first Q value and the current second Q value, the future rewards of the terminal state are shielded by (1-dones) to prevent the incorrect propagation of Q values, thereby improving the accuracy of the trained target control model.

[0113] Optionally, the initial value main network and the initial value target network in this embodiment can pass through two or three fully connected layers (hidden layer dimension 64) to process the continuous state and action space, and ultimately output the current first Q value or the current second Q value. The activation function of the initial value main network and the initial value target network can be a ReLU activation function.

[0114] Step 405: Determine the smaller value between the current first Q value and the current second Q value as the current target Q value.

[0115] In step 405, in order to improve the accuracy of the trained target control model, the smaller value between the current first Q value and the current second Q value is determined as the current target Q value, that is, the worse value between the current first Q value and the current second Q value is determined as the current target Q value to achieve subsequent back propagation.

[0116] Step 406: If the iteration termination condition is not met, the initial value network is updated according to the current target Q value to obtain an updated initial value network, the initial action network is updated to obtain an updated initial action network, and the process returns to step 401.

[0117] The following describes how to update the initial value network. In step 406, the initial value network is updated according to the current target Q value to obtain an updated initial value network, which may include the following steps 4061a to 4065a.

[0118] Step 4061a: If the current target Q value is the current first Q value, determine the first update weight according to the current first Q value and the initial value main network.

[0119] In step 4061a, a first updated weight is determined based on the current first Q value and the initial value main network through gradient descent. Optionally, in this embodiment, gradient truncation can be used during the gradient descent process to avoid gradient explosion.

[0120] Step 4062a: Determine a second updated weight of the initial value main network based on the first updated weight and the current weight of the initial value target network, and obtain an updated initial value main network based on the second updated weight.

[0121] In step 4062a, a soft update of the weights of the initial value main network is implemented. Specifically, a second updated weight of the initial value main network is determined based on the first updated weight and the current weight of the initial value target network. For example, the second updated weight may be the sum of the product of the first updated weight and the corresponding weight coefficient and the product of the current weight of the initial value target network and the corresponding weight coefficient. After obtaining the second updated weight, the initial value main network is updated based on the second updated weight to obtain an updated initial value main network.

[0122] Step 4063a: Determine a third update weight of the initial value target network based on the current first Q value and the initial value target network, and obtain an updated initial value target network based on the third update weight.

[0123] In step 4063a, the initial value target network is updated according to the current first Q value and the initial value target network to obtain an updated initial value target network.

[0124] Step 4064a: If the current target Q value is the current second Q value, determine a fourth update weight based on the current second Q value and the initial value main network, and obtain an updated initial value main network based on the fourth update weight.

[0125] Step 4065a: Determine a fifth update weight according to the current first Q value and the initial value target network, and obtain an updated initial value target network according to the fifth update weight.

[0126] In steps 4064a and 4065a, if the current second Q-value is lower than the current first Q-value (i.e., if the current second Q-value is poor), the probability of drastic fluctuations in the output of the initial value main network is low. Therefore, there is no need to soft-update the weights of the initial value main network. In step 4064a, a fourth update weight is determined directly based on the current second Q-value and the initial value main network, and an updated initial value main network is obtained based on the fourth update weight. In step 4065a, a fifth update weight is determined based on the current first Q-value and the initial value target network, and an updated initial value target network is obtained based on the fifth update weight.

[0127] Steps 4061a to 4065a described above can flexibly select whether to soft-update the initial value main network based on different implementations of the current Q value. This improves training stability when a soft-update is required, and improves training efficiency when a soft-update is not required. Therefore, the above implementation method balances training stability and training efficiency.

[0128] The following describes how to update the initial mobile network. In step 406, the initial mobile network is updated to obtain an updated initial mobile network, which may include the following steps 4061b to 4064b.

[0129] Step 4061b: Determine the sixth updated weight according to the current target Q value and the initial action target network, and obtain the updated initial action target network according to the sixth updated weight.

[0130] Step 4062b: Determine the seventh update weight according to the current target Q value and the initial action main network.

[0131] Step 4063b: Determine the eighth updated weight according to the sixth updated weight and the seventh updated weight.

[0132] Step 4064b: Obtain an updated initial action main network based on the eighth updated weight.

[0133] In steps 4062b to 4064b, a soft update of the initial action master network is performed to improve the stability of the training process. Optionally, the sum of the product of the sixth update weight and the corresponding weight coefficient and the product of the seventh update weight and the corresponding weight coefficient can be determined as the eighth update weight. For example, the weight coefficient corresponding to the seventh update weight can be 0.005. The weight coefficient corresponding to the sixth update weight can be in the range of 1-0.005. By adjusting the weight coefficients corresponding to the sixth update weight and the seventh update weight, the update speed of the initial action master network can be controlled.

[0134] In subsequent iterations, the initial action main network in an iteration is the updated initial action main network in the previous iteration, the initial action target network is the updated initial action target network in the previous iteration, the initial value main network in an iteration is the updated initial value main network in the previous iteration, and the initial value target network is the updated initial value target network in the previous iteration.

[0135] Optionally, in this embodiment, the learning rate of the value network may be 1e-3, and the learning rate of the action network may be 1e-4.

[0136] Step 407: If the iteration termination condition is met, the initial action network is determined as the target control model.

[0137] The target control model is used to determine the current action of the input controlled device according to the current state of the controlled device.

[0138] In step 407 , the initial action network refers to the initial value main network, that is, when the iteration termination condition is met, the initial value main network is determined as the target control model.

[0139] The above model training method is suitable for control in continuous action space. It adopts a dual network structure of main network and target network to improve the stability of the training process and realizes efficient policy gradient optimization through action network-value network.

[0140] Figure 5 Schematic diagram of each model provided by the embodiment of the present invention. Figure 5As shown, in this model training method, the sample generation model inputs the first type of training samples, and the controlled device generates the second type of training samples after actual operation. Furthermore, to further improve training efficiency, each training sample can be preprocessed by normalization and missing value filling. Based on the two types of training samples in the training sample buffer, model training is performed using the action network-value network model. During training, based on the current target Q value, the initial action main network uses a soft update method. When conditions are met, the initial value main network uses a soft update method to improve the stability of the training process. When the iteration termination conditions are met, the initial action main network is determined as the target control model. The target control model can control the controlled device online.

[0141] Figure 5 The figure shows how to implement control based on a target control model in a scenario where the controlled device includes a PI controller, a power source, and an accelerator. The following example shows how to implement control based on a target control model: Based on the current desired parameters and the accelerator's current output parameters obtained through the pick-up coupling loop, the current state (the current error and the integral of the current error) is obtained. Based on the current state, the target control model outputs the current action of the input PI controller: the current integral parameter and the current proportional parameter. Based on the current integral parameter, the current proportional parameter, the current error, and the integral of the current error, the PI controller obtains the control parameters of the input power source. The power source, under the control of the control parameters, feeds an RF signal into the accelerator cavity. This control process repeats itself to achieve real-time and precise control of the controlled device.

[0142] The model training method provided in this embodiment adopts a dual network architecture of a main network and a target network to update the initial value network and the initial action network according to the smaller value of the current first Q value and the current second Q value, thereby improving the accuracy of the trained target control model.

[0143] Figure 6 This is a flow chart of the control method provided by an embodiment of the present invention. This embodiment is applicable to the scenario of controlling the controlled device. The method can be executed by a control device, which can be implemented in the form of hardware and / or software. The control device can be configured in an electronic device, for example, a computer device. Figure 6 As shown, the method includes the following steps 601 to 603.

[0144] Step 601: Obtain the current status of the controlled device.

[0145] Optionally, the controlled device in this embodiment includes a PI controller, a power source for outputting a radio frequency signal, and a radio frequency signal receiving device connected in sequence. Further, the radio frequency signal receiving device is an accelerator.

[0146] Optionally, the current state includes: a current error and an integral of the current error.

[0147] Step 602: Input the current state into the target control model to obtain the current action output by the target control model.

[0148] The target control model is a target control model obtained by the model training method provided in any of the above embodiments.

[0149] Optionally, in a scenario where the controlled device includes a PI controller, a power source for outputting a radio frequency signal, and a radio frequency signal receiving device connected in sequence, the current action includes: a current proportional parameter and a current integral parameter.

[0150] Step 603: Input the current action to the controlled device.

[0151] The controlled device can perform corresponding actions according to the current action. In the scenario where the controlled device includes a PI controller, a power source for outputting RF signals, and an RF signal receiving device connected in sequence, such as Figure 2 and Figure 5 As shown in Figure 1, the PI controller calculates the control parameters for the input power source based on the current integral parameter, the current proportional parameter, the current error, and the integral of the current error. The power source then feeds an RF signal into the accelerator cavity under the control of these control parameters. This control process repeats itself, achieving real-time and precise control of the controlled device.

[0152] Figure 7 It is a schematic diagram of the response curve of the control system. Figure 7 The horizontal axis is time, and the vertical axis represents the output parameter, proportional parameter, integral parameter and error. Figure 7 As shown, the system error converges in a very short time, and the output parameters are relatively stable relative to the step response (i.e. Figure 7 The overshoot (the red line segment in the top figure) is very low. When the target control model obtained by the model training method provided by this embodiment is used to control the controlled device, the optimal proportional and integral parameters of the input PI controller can be found in a relatively short time.

[0153] The model training method provided in this embodiment transforms the parameter variation law of the time-varying parameter system into the action network-value network model architecture and physical model parameters, reflecting the law of parameter variation with state. Through its application in the accelerator low-level control system, Figure 7 The results shown demonstrate the effectiveness of this reinforcement learning plus traditional control algorithm.

[0154] The control method provided in this embodiment obtains the current state of the controlled device, inputs the current state into the target control model, and obtains the current action output by the target control model, wherein the target control model is the target control model obtained according to the above-mentioned model training method, and the current action is input into the controlled device, thereby realizing precise control of the controlled device using the target control model.

[0155] Figure 8 This is a schematic diagram of the structure of a model training device provided by an embodiment of the present invention. Figure 8 As shown, the model training device provided by this embodiment includes: a first acquisition module 81, a first determination module 82, a second determination module 83, an update module 84 and a third determination module 85.

[0156] The first acquisition module 81 is used to acquire a training sample set.

[0157] The training sample set includes a first type of training samples and a second type of training samples. The first type of training samples are data generated by a sample generation model, and the second type of training samples are data generated after the controlled device is actually running.

[0158] The first determination module 82 is configured to input the next training state in the training sample into the initial action network to obtain the next training action output by the initial action network.

[0159] The second determination module 83 is used to input the next training action, the current training state in the training sample, the current training action and the next training state into the initial value network to obtain the current target Q value output by the initial value network.

[0160] The updating module 84 is configured to update the initial value network according to the current target Q value to obtain an updated initial value network, update the initial action network to obtain an updated initial action network, and return to execute the steps executed by the first acquisition module 81 if the iteration termination condition is not met.

[0161] The third determining module 85 is configured to determine the initial action network as the target control model if the iteration termination condition is met.

[0162] The target control model is used to determine the current action input to the controlled device according to the current state of the controlled device.

[0163] In one embodiment, the initial action network includes an initial action main network and an initial action target network. The first determination module 82 is specifically configured to input the next training state from different training samples into the initial action main network and the initial action target network, respectively, to obtain the next main training action output by the initial action main network and the next target training action output by the initial action target network.

[0164] In one embodiment, the initial value network includes an initial value main network and an initial value target network. The second determination module 83 is specifically configured to: input the next training action and the next training state into the initial value target network to obtain a current first Q value output by the initial value target network, wherein the current first Q value is a value determined based on the next training action, the next training state, and the reward value in the training sample, and the next training action includes the next main training action and the next target training action; input the current training action and the current training state into the initial value main network to obtain a current second Q value output by the initial value main network, wherein the current second Q value is a value determined based on the current training action, the current training state, and the reward value in the training sample; and determine the smaller of the current first Q value and the current second Q value as the current target Q value.

[0165] In one embodiment, in terms of updating the initial value network according to the current target Q value to obtain an updated initial value network, the updating module 84 is specifically used to: if the current target Q value is the current first Q value, determine a first update weight according to the current first Q value and the initial value main network; determine a second update weight of the initial value main network according to the first update weight and the current weight of the initial value target network, and obtain an updated initial value main network according to the second update weight; determine a third update weight of the initial value target network according to the current first Q value and the initial value target network, and obtain an updated initial value target network according to the third update weight; if the current target Q value is the current second Q value, determine a fourth update weight according to the current second Q value and the initial value main network, and obtain an updated initial value main network according to the fourth update weight; determine a fifth update weight according to the current first Q value and the initial value target network, and obtain an updated initial value target network according to the fifth update weight.

[0166] In one embodiment, in terms of updating the initial action network to obtain an updated initial action network, the updating module 84 is specifically configured to: determine a sixth update weight based on the current target Q value and the initial action target network, and obtain the updated initial action target network based on the sixth update weight; determine a seventh update weight based on the current target Q value and the initial action main network; determine an eighth update weight based on the sixth update weight and the seventh update weight; and obtain the updated initial action main network based on the eighth update weight.

[0167] In one embodiment, the controlled device includes: a PI controller, a power source for outputting a radio frequency signal, and a radio frequency signal receiving device connected in sequence. The current action includes: a current proportional parameter and a current integral parameter. The current state includes: a current error and an integral of the current error.

[0168] The model training device provided in the embodiment of the present invention can execute the model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0169] Figure 9 FIG. 1 is a schematic diagram of the structure of a control device provided by an embodiment of the present invention. Figure 9 As shown, the control device provided in this embodiment includes: a second acquisition module 91 , a first input module 92 and a second input module 93 .

[0170] The second acquisition module 91 is used to acquire the current state of the controlled device.

[0171] The first input module 92 is used to input the current state into the target control model to obtain the current action output by the target control model.

[0172] The target control model is a target control model obtained according to the model training method in any of the above embodiments.

[0173] The second input module 93 is configured to input the current action into the controlled device.

[0174] The control device provided in the embodiment of the present invention can execute the control method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0175] Figure 101 is a block diagram of an electronic device that implements the control method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided for example only and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0176] like Figure 10 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0177] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0178] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the model training method or the control method.

[0179] In some embodiments, the model training method or the control method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the model training method or the control method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the model training method or the control method in any other appropriate manner (e.g., by means of firmware).

[0180] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0182] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0183] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0184] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0185] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0186] An embodiment of the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the model training method or control method provided in any embodiment of the present invention.

[0187] The computer program product may be implemented in a computer program code for performing the operations of the present invention written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​and conventional procedural programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0188] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0189] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A model training method, characterized in that: The method comprises: Obtaining a training sample set; wherein the training sample set includes a first type of training samples and a second type of training samples, the first type of training samples are data generated by a sample generation model, and the second type of training samples are data generated after the controlled device is actually operated; Inputting the next training state in the training sample into the initial action network to obtain the next training action output by the initial action network; Inputting the next training action, the current training state in the training sample, the current training action and the next training state into an initial value network to obtain a current target Q value output by the initial value network; If the iteration termination condition is not met, then the initial value network is updated according to the current target Q value to obtain an updated initial value network, the initial action network is updated to obtain an updated initial action network, and the process returns to the step of "obtaining a training sample set"; If the iteration termination condition is met, the initial action network is determined as a target control model; wherein the target control model is used to determine a current action input to the controlled device according to the current state of the controlled device.

2. The method according to claim 1, characterized in that The initial action network includes: an initial action main network and an initial action target network; The step of inputting the next training state in the training sample into the initial action network to obtain the next training action output by the initial action network includes: The next training states in different training samples are respectively input into the initial action main network and the initial action target network to obtain the next main training action output by the initial action main network and the next target training action output by the initial action target network.

3. The method according to claim 2, characterized in that The initial value network includes an initial value main network and an initial value target network; The step of inputting the next training action, the current training state in the training sample, the current training action, and the next training state into an initial value network to obtain a current target Q value output by the initial value network includes: Inputting the next training action and the next training state into the initial value target network to obtain a current first Q value output by the initial value target network; wherein the current first Q value is a value determined according to the next training action, the next training state, and the reward value in the training sample, and the next training action includes a next main training action and a next target training action; Inputting the current training action and the current training state into the initial value main network to obtain a current second Q value output by the initial value main network; wherein the current second Q value is a value determined according to the current training action, the current training state, and the reward value in the training sample; A smaller value between the current first Q value and the current second Q value is determined as the current target Q value.

4. The method according to claim 3, characterized in that The updating of the initial value network according to the current target Q value to obtain an updated initial value network includes: If the current target Q value is the current first Q value, determining a first update weight according to the current first Q value and the initial value main network; Determining a second updated weight of the initial value main network according to the first updated weight and the current weight of the initial value target network, and obtaining an updated initial value main network according to the second updated weight; Determining a third updated weight of the initial value target network according to the current first Q value and the initial value target network, and obtaining an updated initial value target network according to the third updated weight; If the current target Q value is the current second Q value, determining a fourth update weight according to the current second Q value and the initial value main network, and obtaining an updated initial value main network according to the fourth update weight; A fifth update weight is determined according to the current first Q value and the initial value target network, and an updated initial value target network is obtained according to the fifth update weight.

5. The method according to claim 2, characterized in that The updating of the initial action network to obtain an updated initial action network includes: Determining a sixth updated weight according to the current target Q value and the initial action target network, and obtaining an updated initial action target network according to the sixth updated weight; Determining a seventh update weight based on the current target Q value and the initial action main network; Determining an eighth updated weight according to the sixth updated weight and the seventh updated weight; An updated initial action main network is obtained according to the eighth updated weight.

6. The method according to any one of claims 1 to 5, characterized in that The controlled device includes: a proportional-integral (PI) controller, a power source for outputting a radio frequency signal, and a radio frequency signal receiving device connected in sequence; The current action includes: a current proportional parameter and a current integral parameter; The current state includes: a current error and an integral of the current error.

7. A control method, characterized in that: The method comprises: Get the current status of the controlled device; Inputting the current state into a target control model to obtain a current action output by the target control model; wherein the target control model is a target control model obtained according to the model training method according to any one of claims 1 to 6; The current action is input into the controlled device.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the model training method described in any one of claims 1 to 6, or execute the control method described in claim 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is used to enable a processor to implement the model training method described in any one of claims 1 to 6, or execute the control method described in claim 7 when executed.

10. A computer program product comprising a computer program, characterized in that When executed by a processor, the computer program implements the model training method according to any one of claims 1 to 6, or executes the control method according to claim 7.

Images