Motion control method and apparatus

Abstract

A motion control method (100) includes determining a motion control model (211) of a controlled object, training an online reinforcement learning model (223) according to the motion control model (211), the motion control model (211) outputting a model control value, the controlled object generating a feedback value (110) according to the model control value and an initial control value output by the online reinforcement learning model (223), calculating a reward (120) using the model control value and the feedback value, the online reinforcement learning model (223) generating a residual control value according to the reward, the model control value and the feedback value, and controlling motion (130) of the controlled object according to the residual control value and the model control value.

Description

Technical Field

[0001] This invention relates primarily to the field of motion control, and more particularly to a motion control method and apparatus. Background Technology

[0002] Motion control optimization is crucial for improving the performance of product production lines. For example, optimizing the speed control of servo motors and the synchronous multi-axis position control can significantly improve the performance of product production lines.

[0003] Motion control optimization is usually achieved by selecting model parameters. The dynamic model of the controlled object (e.g., the actuator) is designed by experienced domain experts. Selecting the appropriate model and optimizing the model parameters will result in better motion control performance. However, this manual optimization process is time-consuming, labor-intensive, and inefficient.

[0004] To overcome the shortcomings of manual optimization, reinforcement learning has been introduced to learn the optimal parameters in motion control models. Such reinforcement learning can achieve automated optimization. However, modeling the controlled object requires deep domain knowledge and is limited by the performance improvement effect of the motion model itself. In addition, reinforcement learning models can only be applied to the current controlled object and cannot be reused to other controlled objects. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a motion control method and apparatus, and improves the efficiency of reinforcement learning modeling in motion control.

[0006] To achieve the above objectives, this invention proposes a motion control method, comprising: determining a motion control model of a controlled object; training an online reinforcement learning model based on the motion control model; the motion control model outputting a model control value; the controlled object generating a feedback value based on the model control value and the initial control value output by the online reinforcement learning model; calculating a reward using the model control value and the feedback value; the online reinforcement learning model generating a residual control value based on the reward, the model control value, and the feedback value; and controlling the motion of the controlled object based on the residual control value and the model control value. Therefore, the online reinforcement learning model is trained based on the motion control model, eliminating the need for training from scratch and improving the training efficiency of the online reinforcement learning model.

[0007] Preferably, the motion control method includes: sending the motion control model, model control values, feedback values, and rewards to the cloud; training an offline reinforcement learning model based on the motion control model, model control values, feedback values, and rewards; updating the original online reinforcement learning model with the offline reinforcement learning model; or deploying the offline reinforcement learning model to a motion control system that does not have an online reinforcement learning model. Therefore, uploading data collected during the online reinforcement learning process of the controlled object to the cloud for classification and training of an offline reinforcement learning model allows for deployment in motion control systems of the same kinematic type, improving the versatility of reinforcement learning models in motion control.

[0008] Preferably, before updating the online reinforcement learning model to the offline reinforcement learning model, the process includes: obtaining the kinematic type of the controlled object; if the kinematic type of the controlled object matches the kinematic type of the offline reinforcement learning model, updating the original online reinforcement learning model with the offline reinforcement learning model, or deploying the offline reinforcement learning model to a motion control system that does not have an online reinforcement learning model. Therefore, by determining the consistency between the kinematic type of the controlled object and the kinematic type of the offline reinforcement learning model, the targeting of the update or deployment can be improved.

[0009] Preferably, the model control value includes an axis position control value, and the feedback value includes an axis position feedback value. Calculating the reward using the model control value and the feedback value includes: calculating the axis position following error based on the axis position control value and the axis position feedback value, and calculating the reward based on the axis position following error. Therefore, reward calculation using the axis position control value and feedback value is achieved.

[0010] Preferably, determining the motion control model of the controlled object includes: receiving the kinematic type selected by the user and input model parameters, under which the controlled object is activated. Therefore, the user only needs to roughly select the kinematic type and input model parameters, without needing to optimize the parameters, reducing the requirements for user modeling and improving the automation and intelligence of motion control.

[0011] Preferably, after sending the motion control model, model control values, feedback values, and rewards to the cloud, the motion control model, model control values, feedback values, and rewards are classified into multiple training datasets, and the offline reinforcement learning model is trained using these multiple training datasets. This achieves the training of the offline reinforcement learning model.

[0012] This invention also proposes a motion control device, comprising: a determination module for determining a motion control model of a controlled object, training an online reinforcement learning model based on the motion control model, the motion control model outputting a model control value, and the controlled object generating a feedback value based on the model control value and the initial control value output by the online reinforcement learning model; a reward calculation module for calculating a reward using the model control value and the feedback value; and a control module for the online reinforcement learning model generating a residual control value based on the reward, the model control value, and the feedback value, and controlling the motion of the controlled object based on the residual control value and the model control value.

[0013] Preferably, the motion control device includes: sending the motion control model, model control value, feedback value, and reward to the cloud; training an offline reinforcement learning model based on the motion control model, model control value, feedback value, and reward; updating the original online reinforcement learning model with the offline reinforcement learning model; or deploying the offline reinforcement learning model to a motion control system that does not have an online reinforcement learning model.

[0014] Preferably, before updating the online reinforcement learning model to the offline reinforcement learning model, the process includes: obtaining the kinematic type of the controlled object; when the kinematic type of the controlled object is consistent with the kinematic type of the offline reinforcement learning model, updating the original online reinforcement learning model with the offline reinforcement learning model, or deploying the offline reinforcement learning model to a motion control system that does not have an online reinforcement learning model.

[0015] Preferably, the model control value includes an axis position control value, the feedback value includes an axis position feedback value, and calculating the reward using the model control value and the feedback value includes: calculating the axis position following error based on the axis position control value and the axis position feedback value, and calculating the reward based on the axis position following error.

[0016] Preferably, determining the motion control model of the controlled object includes: receiving the kinematic type selected by the user and the input model parameters, and starting the controlled object under the kinematic type and model parameters.

[0017] Preferably, after sending the motion control model, model control value, feedback value, and reward to the cloud, the motion control model, model control value, feedback value, and reward are classified into multiple training datasets, and the offline reinforcement learning model is trained using the multiple training datasets.

[0018] The present invention also proposes an electronic device including a processor, a memory and instructions stored in the memory, wherein the instructions, when executed by the processor, implement the method described above.

[0019] The present invention also proposes a computer-readable storage medium having computer instructions stored thereon, which, when executed, perform the methods described above. Attached Figure Description

[0020] The following figures are intended only to illustrate and explain the invention and do not limit the scope of the invention.

[0021] Figure 1 This is a flowchart of a motion control method according to an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of a motion control method according to an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of a motion control device according to an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention.

[0025] Explanation of reference numerals in the attached figures

[0026] 100 Motion Control Methods

[0027] Steps 110-130

[0028] 210 controller

[0029] 211 Motion Control Model

[0030] 212 transceiver

[0031] 220 edge devices

[0032] 221 Data Acquisition Module

[0033] 222 Reward Calculation Module

[0034] 223 Online Reinforcement Learning Model

[0035] 230 Cloud

[0036] 231 Training Data Processing Module

[0037] 232 training dataset

[0038] 233 Offline Reinforcement Learning Model

[0039] 300 motion control device

[0040] 310 Determine Module

[0041] 320 Reward Calculation Module

[0042] 330 control module

[0043] 400 electronic devices

[0044] 410 processor

[0045] 420 memory Detailed Implementation

[0046] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described with reference to the accompanying drawings.

[0047] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the invention is not limited to the specific embodiments disclosed below.

[0048] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0049] Figure 1 This is a flowchart of a motion control method according to an embodiment of the present invention. Figure 1 As shown, the motion control method 100 includes:

[0050] Step 110: Determine the motion control model of the controlled object, train an online reinforcement learning model based on the motion control model, the motion control model outputs a model control value, and the controlled object generates a feedback value based on the model control value and the initial control value output by the online reinforcement learning model.

[0051] In some embodiments, determining the motion control model of the controlled object includes: receiving a user-selected kinematic type and input model parameters, and starting the controlled object under the kinematic type and model parameters. Therefore, the user only needs to adjust the model parameters to enable the controlled object to start, without requiring parameter optimization, significantly reducing workload and improving the efficiency of motion control. The kinematic type can be designed or selected by the user based on the kinematic or application requirements of the controlled object (e.g., a single-axis drive or a synchronous multi-axis drive). For example, proportional-integral-derivative (PID) control is selected for a single-axis drive, or Cartesian position control is selected for a synchronous multi-axis drive. Thus, by selecting the kinematic type of the motion control model and initially inputting the parameters of the motion control model, the motion control model of the controlled object can be determined, which outputs a model control value U. mFurthermore, an online reinforcement learning model is trained based on the motion control model. The online reinforcement learning model outputs an initial control value U at the initial time step. a0 The controlled object is based on the model control value U m and initial control value U a0 It performs motion and generates a feedback value during the motion. The feedback value can be a shaft position value, shaft speed value, shaft torque value, etc.

[0052] Figure 2 This is a schematic diagram of a motion control method according to an embodiment of the present invention. Figure 2 Multiple motion control systems A, B, and C are illustrated, each including a control device and a controlled object. Taking motion control system A as an example, the control device includes an edge device 220 and a controller 210, and the controlled object is a driver 240, which can drive a motor to rotate. It is understood that the control device in the embodiments of the present invention is not limited to this; the control device can also be a single hardware component. The edge device 220 can be an industrial computer (IPC), and the controller 210 can be a programmable logic controller (PLC). In other embodiments, the control device can be an industrial computer with an internal virtual PLC, or the control device can be a PLC with an integrated computing module. Figure 2 As shown, the controller 210 includes a motion control model 211. The motion control model 211 can be selected by the user, with its kinematic type selected, such as PID control or Cartesian position control. The user inputs model parameters, thereby determining the motion control model 211. The motion control model 211 outputs a model control value U. m An online reinforcement learning model is trained based on the motion control model. The online reinforcement learning model outputs an initial control value U at the initial time. a0 The controlled object is based on the model control value U m and initial control value U a0 Engage in exercise and generate a feedback value during the exercise.

[0053] Step 120: Calculate the reward using the model control value and feedback value.

[0054] Based on the motion control model and the actuator kinematics type, rewards can be calculated using model control values ​​and feedback values, such as axis position following error, axis speed following error, axis torque following error, Cartesian position following error, and Cartesian speed following error. In some embodiments, model control values ​​include axis position control values, feedback values ​​include axis position feedback values, and calculating rewards using model control values ​​and feedback values ​​includes: calculating axis position following error based on axis position control values ​​and axis position feedback values, and calculating rewards based on axis position following error.

[0055] For example, for a single-axis drive, the reward can be calculated using the following formula:

[0056] r = 1 / |err pos |

[0057] Where r is the reward, err pos It is the axis position following error, which can be obtained by subtracting the feedback axis position value from the model axis position control value.

[0058] For example, for synchronous multi-axis drives, the reward can be calculated using the following formula:

[0059] r = 1 / ||err x +err y +err z ||

[0060] Where r is the reward, err x err y err z These are the Cartesian position errors in the X, Y, and Z directions, respectively.

[0061] like Figure 2 As shown, the data acquisition module 221 acquires the model control value output by the motion control model 211 and the feedback value generated by the driver 240, and sends the model control value and feedback value to the reward calculation module 222. The reward calculation module 222 calculates the reward based on the model control value and feedback value, and sends the reward to the reinforcement learning model 223.

[0062] Step 130: The online reinforcement learning model generates a residual control value based on the reward, model control value, and feedback value, and controls the movement of the controlled object based on the residual control value and the model control value.

[0063] An online reinforcement learning model can be trained using the SARSA (state-action-reward-state-action) algorithm. The trained online reinforcement learning model generates a residual control value U based on the reward, the model control value, and the feedback value. a According to the residual control value U a and model control value U m Control the movement of the controlled object.

[0064] like Figure 2 As shown, reinforcement learning model 223 receives the reward sent by reward calculation module 222, and receives the model control value and feedback value sent by data acquisition module. Based on the reward, model control value, and feedback value, it generates a residual control value U. a The residual control value is then sent to the transceiver 212 in the controller 210. The transceiver 212 transmits the residual control value U. aand model control value U m The controlled object sends the residual control value U to driver 240. a and model control value U m The system performs motion and continuously generates feedback values ​​during the motion. These feedback values ​​are then sent to the data acquisition module 221, and the process is repeated iteratively until the following error is eliminated, thus achieving the expected control.

[0065] In some embodiments, to improve the versatility of the control model, the motion control method may include sending data (including motion control models, model control values, feedback values, rewards, etc.) collected during online reinforcement learning of multiple control systems to the cloud, and training a general offline reinforcement learning model based on the motion control model, model control values, feedback values, and rewards. The relevant data used to train the offline reinforcement learning model needs to be collected from controlled objects and their control systems that have the same kinematic type.

[0066] In some embodiments, before updating the online reinforcement learning model to the offline reinforcement learning model, the method includes: obtaining the kinematic type of the controlled object; when the kinematic type of the controlled object is consistent with the kinematic type of the offline reinforcement learning model, updating the original online reinforcement learning model with the offline reinforcement learning model, or deploying the offline reinforcement learning model to a motion control system that does not have an online reinforcement learning model.

[0067] In some embodiments, after the motion control model, model control values, feedback values, and rewards are sent to the cloud, the motion control model, model control values, feedback values, and rewards are classified into multiple training datasets, and the offline reinforcement learning model is trained using these multiple training datasets. For example, the CQL algorithm (Conservative Q-Learning algorithm) can be used to train the offline reinforcement learning model.

[0068] Continue to refer to Figure 2As shown, it also includes a cloud platform 230, which includes a training data processing module 231, a training dataset 232, and an offline reinforcement learning model 233. The data acquisition module 231 sends the motion control model, model control values, feedback values, and rewards to the training data processing module 231 of the cloud platform 230. The training data processing module 231 classifies the motion control model, model control values, feedback values, and rewards into multiple training datasets 232 according to kinematic type. Using the multiple training datasets 232, the offline reinforcement learning model 233 is trained using the CQL algorithm (Conservative Q-Learning algorithm), thereby realizing the training of the offline reinforcement learning model 233. The trained offline reinforcement learning model 233 is then updated into the online reinforcement learning model of the motion control system A, or deployed to motion control systems B and C that do not have an online reinforcement learning model, thereby improving the versatility of the reinforcement learning model in motion control. To improve the targeting of deployment or updates, the kinematic type of the controlled object is obtained before deployment or update. If the kinematic type of the controlled object is consistent with the kinematic type of the offline reinforcement learning model 233, the original online reinforcement learning model is updated with the offline reinforcement learning model, or the offline reinforcement learning model is deployed to a motion control system that does not have an online reinforcement learning model.

[0069] Embodiments of this invention provide a motion control method. An online reinforcement learning model is trained based on a motion control model, eliminating the need for retraining from scratch and improving the training efficiency of the online reinforcement learning model. Furthermore, data collected during the online reinforcement learning process of the controlled object is uploaded to the cloud for classification and training to generate an offline reinforcement learning model. This model can be deployed in motion control systems of the same kinematic type, improving the versatility of reinforcement learning models in motion control.

[0070] The present invention also proposes a motion control device. Figure 3 This is a schematic diagram of a motion control device 300 according to an embodiment of the present invention, as shown below. Figure 3 As shown, the motion control device 300 includes:

[0071] Module 310 determines the motion control model of the controlled object, trains an online reinforcement learning model based on the motion control model, outputs a model control value, and the controlled object generates a feedback value based on the model control value and the initial control value output by the online reinforcement learning model.

[0072] The reward calculation module 320 calculates rewards using model control values ​​and feedback values.

[0073] The control module 330 uses an online reinforcement learning model to generate a residual control value based on the reward, model control value, and feedback value. The model control value is used to control the movement of the controlled object.

[0074] In some embodiments, the motion control device 300 includes: sending a motion control model, model control values, feedback values, and rewards to the cloud; training an offline reinforcement learning model based on the motion control model, model control values, feedback values, and rewards; updating an existing online reinforcement learning model with the offline reinforcement learning model; or deploying the offline reinforcement learning model to a motion control system that does not have an online reinforcement learning model.

[0075] In some embodiments, before updating an online reinforcement learning model to an offline reinforcement learning model, the method includes: obtaining the kinematic type of the controlled object; when the kinematic type of the controlled object is consistent with the kinematic type of the offline reinforcement learning model, updating the original online reinforcement learning model with the offline reinforcement learning model, or deploying the offline reinforcement learning model to a motion control system that does not have an online reinforcement learning model.

[0076] In some embodiments, the model control value includes the axis position control value, the feedback value includes the axis position feedback value, and the calculation of the reward using the model control value and the feedback value includes: calculating the axis position following error based on the axis position control value and the axis position feedback value, and calculating the reward based on the axis position following error.

[0077] In some embodiments, determining the motion control model of the controlled object includes: receiving a motion control model selected by a user and input model parameters, and starting the controlled object under the motion control model and model parameters.

[0078] In some embodiments, after the motion control model, model control values, feedback values, and rewards are sent to the cloud, the motion control model, model control values, feedback values, and rewards are classified into multiple training datasets, and the offline reinforcement learning model is trained using the multiple training datasets.

[0079] The present invention also proposes an electronic device 400. Figure 4 This is a schematic diagram of an electronic device 400 according to an embodiment of the present invention. Figure 4 As shown, the electronic device 400 includes a processor 410 and a memory 420. The memory 420 stores instructions, which, when executed by the processor 410, implement the method 100 described above.

[0080] The present invention also proposes a computer-readable storage medium having computer instructions stored thereon, which, when executed, perform the method 100 described above.

[0081] Some aspects of the methods and apparatus of this invention can be executed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The aforementioned hardware or software may be referred to as a "data block," "module," "engine," "unit," "component," or "system." The processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. Furthermore, aspects of this invention may be embodied as computer products residing in one or more computer-readable media, including computer-readable program code. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), optical discs (e.g., compact discs (CDs), digital multifunction discs (DVDs), etc.), smart cards, and flash memory devices (e.g., cards, sticks, key drives, etc.).

[0082] Flowcharts are used herein to illustrate the operations performed by the method according to embodiments of this application. It should be understood that the preceding operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more steps may be removed from them.

[0083] It should be understood that although this specification is described according to various embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0084] The above description is merely an illustrative embodiment of the present invention and is not intended to limit the scope of the invention. Any equivalent changes, modifications, and combinations made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A motion control method (100), characterized in that, The motion control method (100) includes: Determine the motion control model of the controlled object, train an online reinforcement learning model based on the motion control model, the motion control model outputs a model control value, and the controlled object generates a feedback value (110) based on the model control value and the initial control value output by the online reinforcement learning model. The reward (120) is calculated using the model control value and the feedback value; The online reinforcement learning model generates a residual control value based on the reward, the model control value, and the feedback value, and controls the motion of the controlled object based on the residual control value and the model control value (130); The motion control method (100) includes: sending the motion control model, model control value, feedback value, and reward to the cloud; training an offline reinforcement learning model based on the motion control model, model control value, feedback value, and reward; updating the original online reinforcement learning model with the offline reinforcement learning model; or deploying the offline reinforcement learning model to a motion control system that does not have an online reinforcement learning model.

2. The motion control method (100) according to claim 1, characterized in that, Before updating the online reinforcement learning model to the offline reinforcement learning model, the process includes: obtaining the kinematic type of the controlled object; when the kinematic type of the controlled object is consistent with the kinematic type of the offline reinforcement learning model, updating the original online reinforcement learning model with the offline reinforcement learning model, or deploying the offline reinforcement learning model to a motion control system that does not have an online reinforcement learning model.

3. The motion control method (100) according to claim 1, characterized in that, The model control value includes the axis position control value, and the feedback value includes the axis position feedback value. Calculating the reward using the model control value and the feedback value includes: calculating the axis position following error based on the axis position control value and the axis position feedback value, and calculating the reward based on the axis position following error.

4. The motion control method (100) according to claim 1, characterized in that, Determining the motion control model of the controlled object includes: receiving the kinematic type selected by the user and the input model parameters, and starting the controlled object under the kinematic type and model parameters.

5. The motion control method (100) according to claim 1, characterized in that, After sending the motion control model, model control values, feedback values, and rewards to the cloud, the motion control model, model control values, feedback values, and rewards are classified into multiple training datasets, and the offline reinforcement learning model is trained using the multiple training datasets.

6. A motion control device (300), characterized in that, The motion control device (300) includes: The module (310) determines the motion control model of the controlled object, trains an online reinforcement learning model based on the motion control model, outputs a model control value, and the controlled object generates a feedback value based on the model control value and the initial control value output by the online reinforcement learning model. The reward calculation module (320) calculates the reward using the model control value and the feedback value; Control module (330): The online reinforcement learning model generates a residual control value based on the reward, the model control value, and the feedback value, and controls the movement of the controlled object based on the residual control value and the model control value; The motion control device (300) includes: sending the motion control model, model control value, feedback value, and reward to the cloud; training an offline reinforcement learning model based on the motion control model, model control value, feedback value, and reward; updating the original online reinforcement learning model with the offline reinforcement learning model; or deploying the offline reinforcement learning model to a motion control system that does not have an online reinforcement learning model.

7. The motion control device (300) according to claim 6, characterized in that, Before updating the online reinforcement learning model to the offline reinforcement learning model, the process includes: obtaining the kinematic type of the controlled object; when the kinematic type of the controlled object is consistent with the kinematic type of the offline reinforcement learning model, updating the original online reinforcement learning model with the offline reinforcement learning model, or deploying the offline reinforcement learning model to a motion control system that does not have an online reinforcement learning model.

8. The motion control device (300) according to claim 6, characterized in that, The model control value includes the axis position control value, and the feedback value includes the axis position feedback value. Calculating the reward using the model control value and the feedback value includes: calculating the axis position following error based on the axis position control value and the axis position feedback value, and calculating the reward based on the axis position following error.

9. The motion control device (300) according to claim 6, characterized in that, Determining the motion control model of the controlled object includes: receiving the kinematic type selected by the user and the input model parameters, and starting the controlled object under the kinematic type and model parameters.

10. The motion control device (300) according to claim 6, characterized in that, After sending the motion control model, model control values, feedback values, and rewards to the cloud, the motion control model, model control values, feedback values, and rewards are classified into multiple training datasets, and the offline reinforcement learning model is trained using the multiple training datasets.

11. An electronic device (400) comprising a processor (410), a memory (420) and instructions stored in the memory (420), wherein the instructions, when executed by the processor (410), implement the method as claimed in any one of claims 1-5.

12. A computer-readable storage medium having stored thereon computer instructions that, when executed, perform the method according to any one of claims 1-5.

Images