Method, device, storage medium and electronic equipment for controlling multi-axis robot trajectory

Abstract

The application provides a multi-axis robot trajectory control method and device, a storage medium and an electronic device. In the application, a simulation environment including a multi-axis robot dynamics model is constructed, the dynamics model is used to simulate the moment of inertia and load change of each axis of the multi-axis robot; based on the simulation environment, a deep reinforcement learning algorithm is used to train the controller parameters of each axis to obtain the target controller parameters of each axis after training, wherein the deep reinforcement learning algorithm takes the overall trajectory accuracy when the multi-axis robot performs a trajectory task as the optimization target; and the trajectory of the multi-axis robot is controlled based on the target controller parameters. The application solves the problem of low overall trajectory accuracy of the multi-axis robot in the prior art under a variable load working condition.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and more specifically, to a method for controlling the trajectory of a multi-axis robot, a control device for the trajectory of a multi-axis robot, a computer-readable storage medium, and an electronic device. Background Technology

[0002] In the field of industrial automation, the trajectory accuracy of multi-axis robots directly affects product quality and production efficiency. The robot end effector must move strictly according to the predetermined trajectory to ensure the consistency and stability of process quality.

[0003] Existing technologies mainly rely on PID parameter tuning of single-axis motors, which often requires manual adjustment of a large number of parameters. The debugging process is complex and time-consuming, making it difficult to adapt to the rapid deployment requirements under different working conditions, resulting in low trajectory accuracy. Summary of the Invention

[0004] The main objective of this application is to provide a control method, a control device, a computer-readable storage medium, and an electronic device for the trajectory of a multi-axis robot, so as to at least solve the problem of low overall trajectory accuracy of multi-axis robots under variable load conditions in the prior art.

[0005] To achieve the above objectives, according to one aspect of this application, a method for controlling the trajectory of a multi-axis robot is provided, comprising: constructing a simulation environment including a dynamic model of the multi-axis robot, the dynamic model being used to simulate the rotational inertia and load changes of each axis of the multi-axis robot; training the controller parameters of each axis using a deep reinforcement learning algorithm based on the simulation environment to obtain the trained target controller parameters of each axis, wherein the deep reinforcement learning algorithm aims to optimize the overall trajectory accuracy of the multi-axis robot when performing a trajectory task; and controlling the trajectory of the multi-axis robot based on the target controller parameters.

[0006] Optionally, before training the controller parameters of each axis using a deep reinforcement learning algorithm based on the simulation environment, the method further includes: constructing a deep reinforcement learning model including a policy network and an evaluation network, wherein the policy network is used to output controller parameters in the action space according to the state space, and the evaluation network is used to evaluate the action value corresponding to the controller parameters in the action space; determining the state space, action space, and reward function of the deep reinforcement learning model; wherein the state space includes at least the real-time trajectory deviation of each axis motor and the maximum deviation of the overall trajectory; the action space includes the controller parameters of each axis, and the controller parameters include the position loop gain parameter, the speed loop proportional gain parameter, and the speed loop integral time constant of each axis; the reward function uses the maximum deviation of the overall trajectory as the optimization index, and the setting of the reward function includes: giving a reward or penalty based on the comparison result of the maximum deviation of the overall trajectory with a preset threshold, and penalizing the real-time trajectory deviation.

[0007] Optionally, based on the simulation environment, a deep reinforcement learning algorithm is used to train the controller parameters of each axis to obtain the trained target controller parameters of each axis, including: in the simulation environment, inputting the state space into the policy network and outputting the controller parameters in the action space; executing the control action corresponding to the controller parameters in the action space and obtaining a reward signal based on the execution result; iteratively updating the network parameters of the policy network and the evaluation network based on the reward signal until a preset termination condition is met to obtain the controller parameters output by the policy network; and determining the controller parameters output by the policy network as the target controller parameters.

[0008] Optionally, the preset thresholds include a first preset threshold, a second preset threshold, and a third preset threshold. Rewards or penalties are given based on the comparison result between the maximum deviation of the overall trajectory and the preset thresholds, including: giving a first positive reward when the maximum deviation of the overall trajectory is less than or equal to the first preset threshold; giving a second positive reward when the maximum deviation of the overall trajectory is greater than the first preset threshold and less than or equal to the second preset threshold; giving a first negative penalty when the maximum deviation of the overall trajectory is greater than the second preset threshold and less than or equal to the third preset threshold; and giving a second negative penalty when the maximum deviation of the overall trajectory is greater than the third preset threshold, wherein the first preset threshold is less than the second preset threshold, and the second preset threshold is less than the third preset threshold.

[0009] Optionally, the preset termination condition includes at least one of the following: if the maximum deviation of the overall trajectory in a single training round is greater than a third preset threshold, the training of the single training round is terminated and the next training round begins; if the cumulative number of training rounds reaches a preset number of rounds, or the cumulative reward of an average preset number of rounds reaches a preset reward value, the overall training is terminated.

[0010] Optionally, a simulation environment including a multi-axis robot dynamics model is constructed, comprising: acquiring a simulation architecture, the simulation architecture including a dynamics model module and a multi-axis motor control model module of the multi-axis robot; configuring the dynamics model module so that the dynamics model module calculates the moment of inertia and load torque of each axis of the multi-axis robot in the current operating state according to a preset trajectory task; establishing a data connection between the dynamics model module and the multi-axis motor control model module, so that the moment of inertia and load torque calculated by the dynamics model module are input as input variables to the multi-axis motor control model module, so as to simulate the influence of load changes on trajectory control in the simulation environment.

[0011] Optionally, the state space may further include at least one of the position feedback, position error, speed feedback, and speed error of each axis motor.

[0012] According to another aspect of this application, a control device for the trajectory of a multi-axis robot is provided, comprising: a first construction unit for constructing a simulation environment including a dynamic model of the multi-axis robot, the dynamic model being used to simulate the rotational inertia and load changes of each axis of the multi-axis robot; a training unit for training the controller parameters of each axis using a deep reinforcement learning algorithm based on the simulation environment to obtain the target controller parameters of each axis after training, wherein the deep reinforcement learning algorithm aims to optimize the overall trajectory accuracy of the multi-axis robot when performing a trajectory task; and a control unit for controlling the trajectory of the multi-axis robot based on the target controller parameters.

[0013] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any of the multi-axis robot trajectory control methods described above.

[0014] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a control method for executing any of the described multi-axis robot trajectories.

[0015] By applying the technical solution of this application, a simulation environment is constructed that includes a dynamic model of a multi-axis robot. The dynamic model is used to simulate the rotational inertia and load changes of each axis of the multi-axis robot. Based on the simulation environment, a deep reinforcement learning algorithm is used to train the controller parameters of each axis to obtain the target controller parameters after training. The deep reinforcement learning algorithm aims to optimize the overall trajectory accuracy of the multi-axis robot when performing trajectory tasks. The trajectory of the multi-axis robot is controlled based on the target controller parameters. In this solution, by constructing a simulation environment with an integrated dynamic model, the deep reinforcement learning algorithm automatically optimizes and obtains controller parameters that adapt to changes in rotational inertia and load, with the goal of achieving optimal overall trajectory accuracy. This effectively solves the problem of trajectory error accumulation caused by inter-axis coupling and changes in working conditions in multi-axis robots, and improves the trajectory accuracy of multi-axis robots under complex motion paths. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a control method for a multi-axis robot trajectory according to an embodiment of this application is shown.

[0018] Figure 2 A flowchart illustrating a method for controlling the trajectory of a multi-axis robot according to an embodiment of this application is shown.

[0019] Figure 3 A schematic diagram illustrating the trajectory accuracy of a multi-axis robot is shown.

[0020] Figure 4 A schematic diagram of the DDPG algorithm, representing a specific method for controlling the trajectory of a multi-axis robot according to an embodiment of this application, is shown.

[0021] Figure 5 A block diagram of a DDPG-based multi-axis robot trajectory control parameter optimization system is shown, illustrating a specific multi-axis robot trajectory control method according to an embodiment of this application.

[0022] Figure 6 A flowchart of the DDPG process is shown, illustrating a specific method for controlling the trajectory of a multi-axis robot according to an embodiment of this application.

[0023] Figure 7 A structural block diagram of a control device for a multi-axis robot trajectory provided according to an embodiment of this application is shown.

[0024] The above figures include the following reference numerals:

[0025] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] As described in the background section, existing technologies mainly rely on PID parameter tuning of single-axis motors, which often requires manual adjustment of a large number of parameters. The debugging process is complex and time-consuming, making it difficult to adapt to the rapid deployment requirements under different working conditions, resulting in low trajectory accuracy. To solve the problem of low overall trajectory accuracy of multi-axis robots under variable load conditions in existing technologies, embodiments of this application provide a multi-axis robot trajectory control method, a multi-axis robot trajectory control device, a computer-readable storage medium, and an electronic device.

[0030] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0031] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1This is a hardware structure block diagram of a mobile terminal for a multi-axis robot trajectory control method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0032] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the multi-axis robot trajectory control method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] This embodiment provides a method for controlling the trajectory of a multi-axis robot that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] Figure 2 This is a flowchart illustrating a multi-axis robot trajectory control method according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0035] Step S201: Construct a simulation environment including a dynamic model of a multi-axis robot. The dynamic model is used to simulate the rotational inertia and load changes of each axis of the multi-axis robot.

[0036] Specifically, step S201 establishes a high-fidelity virtual training platform to overcome the shortcomings of traditional methods in effectively simulating the complex dynamic characteristics of multi-axis linkages. The constructed simulation environment includes a dynamic model of the multi-axis robot. The core function of this dynamic model is to simulate and analyze the changes in the robot's physical characteristics under different tasks in real time, especially to calculate and output the rotational inertia and load torque of each axis motor in the current operating state. By integrating these dynamic parameters that change dynamically with load and motion state into the simulation environment, the effects of frequent load switching, large fluctuations in inertia, and coupling effects such as Coriolis force and nonlinear friction force on the multi-axis robot in actual operation can be realistically reproduced. This provides a training scenario with realistic physical constraints for subsequent reinforcement learning algorithms, thereby ensuring that the trained parameters are robust in practical applications.

[0037] Step S202: Based on the above simulation environment, a deep reinforcement learning algorithm is used to train the controller parameters of each axis to obtain the target controller parameters of each axis after training. The deep reinforcement learning algorithm takes the overall trajectory accuracy of the multi-axis robot when performing trajectory tasks as the optimization objective.

[0038] Specifically, in the constructed simulation environment, deep reinforcement learning algorithms (such as DDPG) are used to train the controller parameters. The algorithm takes the overall trajectory accuracy (i.e., minimizing the maximum deviation between the actual trajectory of the end effector and the ideal trajectory) when the multi-axis robot performs trajectory tasks as the global optimization objective. The agent continuously tries and fails in the simulation environment, taking the real-time state feedback from the dynamic model (such as position error, velocity error, trajectory deviation, etc.) as input, and automatically outputs the parameter combinations of each axis controller (position loop gain parameter Kp, velocity loop proportional gain parameter Kv, velocity loop integral time constant Ti). Through countless iterations of training, the algorithm can autonomously learn how to coordinate the actions of each axis to minimize the overall trajectory error under complex working conditions such as strong multi-axis coupling and load changes, thereby obtaining a set of globally optimal target controller parameters. This process does not rely on a lot of actual physical debugging, improving debugging efficiency and solving the technical problem that traditional methods are difficult to achieve in terms of multi-axis synchronization accuracy.

[0039] Step S203: Control the trajectory of the multi-axis robot based on the target controller parameters.

[0040] Specifically, after training and converging in a simulation environment to obtain the optimal target controller parameters, these parameters are then distributed and configured into the actual control system of the multi-axis robot (e.g., replacing the original fixed PID parameters). In actual operation, the robot controller utilizes these parameters, optimized through deep reinforcement learning, for real-time closed-loop control. Because these parameters are obtained by fully considering dynamic coupling, load inertia variations, and multi-axis cooperative effects, they can significantly suppress coupling interference between axes when facing high-precision trajectory tasks in actual production (such as gluing and welding), effectively reducing trajectory tracking errors, improving the overall motion accuracy, synchronization performance, and system stability of the multi-axis robot, ultimately achieving the technical effect of improving product quality and production efficiency.

[0041] In this embodiment, by constructing a simulation environment with an integrated dynamic model, a deep reinforcement learning algorithm is used to automatically optimize and obtain controller parameters that adapt to changes in rotational inertia and load, with the goal of achieving the best overall trajectory accuracy. This effectively solves the problem of trajectory error accumulation caused by inter-axis coupling and changes in working conditions in multi-axis robots, and improves the trajectory accuracy of multi-axis robots under complex motion paths.

[0042] In some embodiments of this application, before training the controller parameters of each axis using a deep reinforcement learning algorithm based on the above simulation environment, the method further includes: constructing a deep reinforcement learning model including a policy network and an evaluation network, wherein the policy network is used to output controller parameters in the action space according to the state space, and the evaluation network is used to evaluate the action value corresponding to the controller parameters in the action space; determining the state space, action space, and reward function of the deep reinforcement learning model; wherein the state space includes at least the real-time trajectory deviation of each axis motor and the maximum deviation of the overall trajectory; the action space includes the controller parameters of each axis, and the controller parameters include the position loop gain parameter, the speed loop proportional gain parameter, and the speed loop integral time constant of each axis; the reward function uses the maximum deviation of the overall trajectory as the optimization index, and the setting of the reward function includes: giving a reward or penalty based on the comparison result of the maximum deviation of the overall trajectory with a preset threshold, and penalizing the real-time trajectory deviation.

[0043] Specifically, before training with deep reinforcement learning algorithms in a simulation environment, a deep reinforcement learning model must first be constructed. This model mainly consists of a policy network (Actor) and an evaluation network (Critic). The policy network serves as the decision-making core of the agent, taking the current system's state space as input and outputting the combination of controller parameters in the action space. The evaluation network is used to evaluate the value of the actions output by the policy network, that is, to predict the cumulative reward that the agent can obtain in the future after taking a certain set of controller parameters in a specific state. This guides the updating and optimization of the policy network, ensuring the convergence and stability of the parameter optimization process.

[0044] Furthermore, it is necessary to define the state space, action space, and reward function of the deep reinforcement learning model to construct a complete training environment. The state space includes at least key indicators reflecting the robot's real-time motion performance, namely the real-time trajectory deviation of each axis motor and the maximum deviation of the overall trajectory. These indicators can intuitively reflect the difference between the current control effect and the target trajectory. The action space is defined as the set of controller parameters to be optimized for each axis. For a multi-axis robot using three-loop control, the parameters that need to be adjusted for each axis include the position loop gain parameter (Kp), the velocity loop proportional gain parameter (Kv), and the velocity loop integral time constant (Ti). The action space is composed of these parameters for all axes. The reward function uses the maximum deviation of the overall trajectory as the core optimization indicator, aiming to guide the agent to minimize the trajectory error.

[0045] In setting the reward function, a reward and punishment mechanism based on a preset threshold is adopted to accelerate convergence and ensure control accuracy. This mechanism rewards or punishes the agent by comparing the maximum deviation of the overall trajectory with the preset threshold.

[0046] Furthermore, the aforementioned preset thresholds include a first preset threshold, a second preset threshold, and a third preset threshold. Rewards or penalties are given based on the comparison result between the maximum deviation of the overall trajectory and the preset thresholds, including: granting a first positive reward when the maximum deviation of the overall trajectory is less than or equal to the first preset threshold; granting a second positive reward when the maximum deviation of the overall trajectory is greater than the first preset threshold but less than or equal to the second preset threshold; granting a first negative penalty when the maximum deviation of the overall trajectory is greater than the second preset threshold but less than or equal to the third preset threshold; and granting a second negative penalty when the maximum deviation of the overall trajectory is greater than the third preset threshold, wherein the first preset threshold is less than the second preset threshold, and the second preset threshold is less than the third preset threshold.

[0047] Specifically, the preset thresholds in the reward function are divided into a first preset threshold, a second preset threshold, and a third preset threshold, satisfying the relationship that the first preset threshold is less than the second preset threshold, and the second preset threshold is less than the third preset threshold. Through this hierarchical threshold setting, the reward mechanism can provide the agent with feedback signals of varying intensities based on different ranges of the maximum deviation of the overall trajectory, thereby achieving a balance between exploring optimal parameters and maintaining basic control performance.

[0048] When the maximum deviation of the overall trajectory is less than or equal to the first preset threshold, it indicates that the robot's current trajectory control accuracy is extremely high, reaching an ideal level. At this time, the agent will receive the first positive reward (e.g., a high positive reward) to encourage the policy network to maintain or further optimize this high-performance state. When the maximum deviation of the overall trajectory is greater than the first preset threshold but less than or equal to the second preset threshold, it indicates that the trajectory accuracy is in an acceptable but not optimal range. At this time, the agent will receive the second positive reward (e.g., a small positive reward or a low positive reward), which affirms its basic performance while indicating that there is still room for improvement.

[0049] When the maximum deviation of the overall trajectory exceeds the second preset threshold but is less than or equal to the third preset threshold, it indicates that the trajectory accuracy has deviated significantly and is in a critical or undesirable range requiring improvement. In this case, the agent will be subject to a first negative penalty (e.g., a low-level negative penalty) to warn the policy network that the parameter combination is defective. If the maximum deviation of the overall trajectory exceeds the third preset threshold, it indicates that the trajectory control is severely inaccurate and cannot meet the basic operational requirements. In this case, the agent will be subject to a second negative penalty (e.g., a large-scale negative penalty).

[0050] By setting preset threshold ranges, a tiered reward and punishment feedback mechanism is constructed, enabling the deep reinforcement learning agent to receive differentiated signal feedback based on the degree of trajectory deviation. High rewards are given when the deviation is extremely small to reinforce the optimal policy, low rewards are given when the deviation is moderate to preserve performance, and penalties of varying degrees are applied when the deviation exceeds a safe range to suppress inferior policies. This mechanism effectively guides the algorithm to converge quickly to a high-precision control region, avoiding training oscillations or slow convergence caused by a single reward signal, thereby improving the efficiency and accuracy of multi-axis robot trajectory control parameter optimization.

[0051] By constructing a deep reinforcement learning model including a policy network and an evaluation network, and defining a state space with real-time trajectory deviation and maximum overall trajectory deviation as core indicators, an action space including three closed-loop key gain parameters, and a reward function based on threshold rewards and penalties, automated and intelligent optimization of PID parameters for each axis of a multi-axis robot is achieved. Leveraging the adaptive characteristics of reinforcement learning, it can automatically balance the control gains of each axis to minimize trajectory errors without requiring manual experience-based tuning, thus improving trajectory accuracy and parameter tuning efficiency under multi-axis linkage control.

[0052] In some embodiments of this application, the state space further includes at least one of the position feedback, position error, speed feedback, and speed error of each axis motor.

[0053] Specifically, in addition to the two core indicators of real-time trajectory deviation and maximum overall trajectory deviation, the state space also includes at least one of the following: position feedback, position error, speed feedback, and speed error of each axis motor. These multi-dimensional state variables allow for a deeper understanding of the motion details within each axis, going beyond just the final trajectory result. Position feedback and position error reflect the static deviation between the commanded position and the actual position of each axis, serving as direct indicators of position control accuracy. Speed ​​feedback and speed error, on the other hand, reflect the dynamic response characteristics of each axis, which are crucial for suppressing overshoot and oscillation, and improving system stability.

[0054] By introducing the aforementioned state variables, deep reinforcement learning agents can gain a more comprehensive understanding of the coupling effects and dynamic characteristics of multi-axis systems. In multi-axis linkage control, changes in the motion state of one axis can affect other axes through mechanical coupling, thereby influencing the overall trajectory. For example, abnormal fluctuations in velocity error may foreshadow impending trajectory deviation, while the accumulation of position error reflects long-term static deviations in the system. By simultaneously observing these internal state variables and external trajectory deviations, the agent can more accurately determine the effectiveness of the current control strategy, thus selecting a more suitable combination of PID parameters in the action space. This state awareness helps the agent identify potential control defects earlier during training, accelerates convergence, and improves the generalization ability of the final strategy under complex conditions.

[0055] This embodiment enables deep reinforcement learning agents to more keenly capture the coupling effects and dynamic response characteristics (such as overshoot and oscillation tendency) within multi-axis systems. As a result, during the parameter optimization process, it not only optimizes the final trajectory accuracy but also improves the dynamic stability and robustness of the system, accelerates the convergence speed, and enhances the model's generalization ability under complex conditions.

[0056] In some embodiments of this application, based on the above simulation environment, a deep reinforcement learning algorithm is used to train the controller parameters of each axis to obtain the target controller parameters after training for each axis. This includes: in the above simulation environment, inputting the state space into the policy network and outputting the controller parameters in the action space; executing the control action corresponding to the controller parameters in the action space and obtaining a reward signal based on the execution result; iteratively updating the network parameters of the policy network and the evaluation network based on the reward signal until a preset termination condition is met to obtain the controller parameters output by the policy network; and determining the controller parameters output by the policy network as the target controller parameters.

[0057] Specifically, the state-space data is first input into the constructed policy network. Based on the current system state (such as position feedback of each axis motor, speed error, real-time trajectory deviation, and the maximum overall trajectory deviation), the policy network outputs the corresponding combination of controller parameters in the motion space through forward computation, namely the position loop gain, speed loop proportional gain, and speed loop integral time constant for each axis. Subsequently, the simulation environment executes these control parameters output by the policy network, driving the virtual multi-axis robot model to move along a predetermined trajectory and recording the system response during execution. Next, a reward signal is calculated based on the execution results. The reward signal is calculated based on a reward function, comprehensively considering the comparison between the maximum deviation of the overall trajectory and a preset threshold, as well as the penalty term for the real-time trajectory deviation, thereby generating a numerical feedback reflecting the quality of the current control strategy. This reward signal is then input into the evaluation network and the policy network to guide the iterative update of the network parameters. The evaluation network assesses the long-term value based on the current state and actions, while the policy network adjusts its own network weight parameters using optimization algorithms such as gradient ascent or descent based on the feedback from the evaluation network and the magnitude of the reward signal, hoping to obtain higher cumulative rewards in future interactions. The process of inputting the state, outputting the action, obtaining the reward, and updating the network parameters is continuously iterated until the preset termination condition is met.

[0058] By utilizing the iterative mechanism of deep reinforcement learning, the policy network can continuously self-correct based on the real-time control effect (reward signal), automatically explore and lock the optimal combination of PID parameters that maximizes trajectory accuracy, thereby eliminating reliance on human experience and improving the automation level, convergence efficiency and final control accuracy of multi-axis robot trajectory control parameter tuning.

[0059] Furthermore, the aforementioned preset termination conditions include at least one of the following: if the maximum deviation of the overall trajectory in a single training round is greater than a third preset threshold, the training of the single training round is terminated and the next training round begins; if the cumulative number of training rounds reaches a preset number of rounds, or the cumulative reward of an average preset number of rounds reaches a preset reward value, the overall training is terminated.

[0060] Specifically, regarding the termination condition for a single training round, if, in a training round within the simulation environment, the robot's actions cause the maximum deviation of the overall trajectory to exceed a third preset threshold, then the control strategy for that round is deemed severely ineffective and unable to meet basic operational requirements. In this case, training for the current round is immediately terminated without waiting for it to end naturally, thus saving computational resources and preventing poor strategies from negatively impacting the model. Subsequently, a new training round automatically begins. This mechanism ensures that poor parameter combinations do not interfere with the model's subsequent learning.

[0061] Two criteria were established for terminating the overall training to meet different convergence requirements. The first criterion is based on the cumulative number of training rounds. When the total number of training rounds reaches a preset maximum value, training is forcibly stopped regardless of whether the reward continues to increase. This aims to control training costs and prevent infinite training loops. The second criterion is based on average reward convergence. The average cumulative reward over the most recent preset number of rounds (e.g., the last 20 rounds) is calculated. When this average reaches a preset reward threshold, it indicates that the agent has consistently obtained high positive feedback, and the policy network has converged to a better state. At this point, overall training is terminated, and the current policy network output is determined as the final target controller parameters. This reward-based convergence termination mechanism ensures that training only ends when the model performance is stable and excellent, thus guaranteeing that the controller parameters deployed on the actual robot have optimal trajectory control performance.

[0062] This embodiment utilizes a single-round bias threshold mechanism to promptly eliminate inferior strategies, preventing invalid training and accelerating iteration; it prevents training from going on indefinitely by accumulating the number of rounds, controlling computational costs; and it ensures policy stability through average reward convergence judgment, ending training only when the model performance reaches its optimal and stable state. These mechanisms work together to improve the training efficiency, resource utilization, and reliability of the final strategy in deep reinforcement learning for tuning parameters of complex multi-axis robots.

[0063] In some embodiments of this application, constructing a simulation environment including a multi-axis robot dynamics model includes: acquiring a simulation architecture, the simulation architecture including a dynamics model module and a multi-axis motor control model module of the multi-axis robot; configuring the dynamics model module so that the dynamics model module calculates the moment of inertia and load torque of each axis of the multi-axis robot in the current operating state according to a preset trajectory task; establishing a data connection between the dynamics model module and the multi-axis motor control model module, so that the moment of inertia and load torque calculated by the dynamics model module are input as input variables to the multi-axis motor control model module, so as to simulate the influence of load changes on trajectory control in the simulation environment.

[0064] Specifically, constructing a high-fidelity multi-axis robot simulation environment is fundamental to achieving efficient training with deep reinforcement learning. First, a simulation architecture is built, comprising a dynamics model module and a multi-axis motor control model module for the multi-axis robot. This architecture not only simulates the robot's mechanical motion characteristics but also integrates the underlying electrical control logic of the motors, thus comprehensively reflecting the entire process from command issuance to mechanical execution. The dynamics model module simulates the robot's physical characteristics under different operating conditions, while the multi-axis motor control model module simulates the motor drive response based on PID control.

[0065] A dynamic model module is configured to enable dynamic simulation of complex working conditions. During the simulation, the dynamic model module calculates the equivalent moment of inertia and load torque of each axis in real time based on the preset trajectory task (such as typical motion paths like straight lines and circular arcs) and the robot's current operating state (such as position and velocity). This calculation takes into account the dynamic changes in the robot's body inertia and the load of the end effector caused by attitude changes, ensuring that the simulation environment can realistically reproduce the inertia fluctuations and load disturbances in actual operation, providing a near-realistic training scenario for reinforcement learning agents.

[0066] A bidirectional data connection is established between the dynamics model module and the multi-axis motor control model module, forming a closed-loop simulation system. The dynamics model module uses the real-time calculated moments of inertia and load torque of each axis as key input variables, dynamically distributing them to the multi-axis motor control model module. In the multi-axis motor control model module, these dynamically changing physical parameters affect the actual response characteristics of the motor, thereby influencing the trajectory control deviation. Through this connection, the simulation environment can simulate the impact of load changes on trajectory control accuracy in real time, enabling the deep reinforcement learning agent to be trained in a complex environment containing various disturbances and uncertainties. This allows it to learn controller parameters with stronger robustness, improving its adaptability in real-world scenarios.

[0067] By constructing a closed-loop simulation architecture integrating the dynamic model module and the multi-axis motor control model module, the dynamic coupling simulation of load torque and rotational inertia on the motor control system is realized. This enables the deep reinforcement learning agent to be trained in a high-fidelity environment containing real physical disturbances (such as inertia changes and load fluctuations), thereby giving the controller stronger robustness and adaptability, and ensuring that the optimized PID parameters can still maintain high-precision trajectory control performance under actual complex working conditions.

[0068] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the multi-axis robot trajectory control method of this application will be described in detail below with reference to specific embodiments.

[0069] This embodiment relates to a specific control method for the trajectory of a multi-axis robot. When adjusting the proportional-integral-derivative (PID) parameters of each axis of the robot, deep reinforcement learning methods (such as Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO)) are used. This embodiment takes the DDPG method as an example and combines it with the robot's dynamics model (motion trajectory, inertia change, load change) to solve the technical problem of insufficient trajectory accuracy in high-precision trajectory control of multi-axis robots caused by dynamic coupling, parameter differences, load disturbances, and the complexity of manual debugging.

[0070] A schematic diagram of the trajectory accuracy of a multi-axis robot can be found here. Figure 3 The ideal straight trajectory represents the robot's running trajectory planned by motion control; the actual straight trajectories 1-4 are the actual running trajectories of the multi-axis robot under different control parameters, used to explain the running performance and trajectory accuracy of the multi-axis robot; the trajectory accuracy is determined by the maximum deviation between the actual straight trajectory and the ideal straight trajectory during operation. The smaller the deviation, the better the trajectory accuracy. Figure 3 In the comparison, the deviations of the actual straight line trajectories 1 to 4 from the ideal straight line trajectories are compared. The results of the deviation comparison are: actual straight line trajectory 4 > actual straight line trajectory 1 > actual straight line trajectory 3 > actual straight line trajectory 2. Therefore, the actual straight line trajectory 2 has the best trajectory accuracy. The trajectory deviations of each actual straight line trajectory from the ideal straight line trajectory can be obtained by the mathematical formula from the midpoint of space to the straight line, which will not be elaborated here.

[0071] Figure 4 This is a schematic diagram of the DDPG algorithm. Figure 4As shown, Environment refers to the application environment of the DDPG algorithm, which in this embodiment refers to the multi-axis robot multi-axis motor control system; Observations are the observation state variables of DDPG, which are sent into the State (state space) as the state variables of DDPG. In this embodiment, it can be a multi-dimensional vector set of indicators affecting trajectory accuracy, such as position feedback, position error, speed feedback, speed error, real-time trajectory deviation, and maximum trajectory deviation of each axis motor; Q Value is the Q value, also called the action value function; Critic is the evaluation network, Critic Loss is the evaluation function loss function, and Rewards is the reward given to the agent; Actor is the policy network, Actor Loss is the policy network loss function, and Actions are the actions of the policy network; Update represents updating the evaluation network and the policy network; The DDPG algorithm is one of the deep reinforcement learning methods, and its specific principles will not be elaborated here.

[0072] Figure 5 This is a block diagram of a multi-axis robot trajectory control parameter optimization system based on DDPG, such as... Figure 5 As shown, the entire framework is a three-closed-loop position control system, which combines the dynamic model to issue the rotational inertia J of each axis of the multi-axis robot and the load T. L The DDPG deep reinforcement learning algorithm was used to train the model with the goal of achieving optimal trajectory accuracy.

[0073] The DDPG Agent is the intelligent agent of the DDPG algorithm. Figure 4This is its algorithm schematic; Observations are the input to the algorithm, and the sampling frequency of the observation states is 1 / 10 of the feedback sampling frequency. They are indicators reflecting the performance of each axis and the overall performance of the multi-axis robot in the environment in which the agent operates, namely, position feedback, position error, velocity feedback, velocity error, real-time trajectory deviation (if the real-time deviation is minimized, the overall maximum deviation will also be small), and maximum trajectory deviation (the maximum trajectory deviation is a key indicator reflecting trajectory accuracy, also called trajectory accuracy); The Award trajectory target reward formula rewards and punishes based on the performance indicators of each axis and the overall performance of the multi-axis robot. Weights are assigned to these performance indicators, with positive rewards given for values ​​exceeding the target value and penalties given for values ​​falling below the target value, allowing the agent to train and learn; Terminate The termination conditions for Judgement are divided into termination conditions for each training round and termination conditions for the completion of overall agent training. For example, if the maximum deviation of the trajectory obtained in each round exceeds a very poor value, the round ends directly and the next round of training begins. The termination condition for the completion of overall agent training can be that the number of training iterations reaches 1000 or the cumulative reward on average over 20 rounds reaches a good set value. The Action output is the parameter Kp required for each axis position loop and the parameters Kv and Ti required for each axis velocity loop. Kp is the position loop proportional gain bandwidth (i.e., position loop gain). The parameters are in Hz (4~10Hz for a four-axis robot), Kv is the speed loop proportional gain bandwidth (i.e., speed loop proportional gain parameter, in Hz, 20~150Hz for a four-axis robot), Ti is the speed loop integral time constant (10~40ms for a four-axis robot), and the value range is obtained based on the range that each axis can be compatible with in the actual operation of the multi-axis robot; the round cumulative reward is the total reward for each round, and the reward formula gives the reward for a single calculation cycle in each round.

[0074] The dynamic model module sends out the rotational inertia J and load T of each axis in real time. L ; It is the electrical angle feedback value of the motor rotor. It is the motor rotor position command value; It is the direct axis (d-axis) reference current. It is the quadrature axis (q-axis) reference current; It is the d-axis feedback current. It is the q-axis feedback current; It is the injected current in the form of a ramp; P position loop is the position loop controller, PI1 is the speed loop proportional-integral controller, and PI2 is the current loop proportional-integral controller. It is the d-axis reference voltage. It is the q-axis reference voltage; It is the inverse Park transform module. This is a coordinate transformation module from the natural coordinate system to the two-phase rotating coordinate system. This is a common motor coordinate transformation module, so it will not be described in detail. It is the reference voltage along the a-axis of the stationary coordinate system. It is the reference voltage of the b-axis in the stationary coordinate system; SVPWM is the space vector modulation algorithm module, which is a common motor vector control module, so it will not be described in detail. This is the DC power supply bus voltage of the inverter module; , , These are the three-phase currents of the permanent magnet synchronous motor; PMSM is the Permanent Magnet Synchronous Motor (PMSM) module.

[0075] The following is a specific example:

[0076] First, in the simulation software MATLAB, according to Figure 3 The principle block diagram is used to build a simulation platform for a multi-axis motor control system. The dynamic model can simulate the operating state under different load conditions and different trajectory tasks, and calculate the rotational inertia J and load T. L The calculated moment of inertia J is compared with the load T. L It is integrated into the simulation environment, and typical trajectory tasks are set as training targets (such as straight-line trajectories).

[0077] Employing deep reinforcement learning algorithms, taking the Deep Deterministic Policy Gradient (DDPG) algorithm as an example, the PID parameters of the position and velocity loops of each axis are used as the action space output values ​​of the agent; combined with... Figure 5 Taking a four-axis robot as an example, its agent motion space output value can be represented as follows: 3 parameters for each axis, for a total of 12 parameters.

[0078] ;

[0079] The state space is defined by performance indicators of each axis that reflect the trajectory accuracy of a multi-axis robot, such as the position feedback of each axis motor. Position error Speed ​​feedback Speed ​​error Real-time trajectory deviation (If the real-time deviation is minimized, the overall maximum deviation will also be small), maximum trajectory deviation (The maximum deviation of the trajectory is a key indicator reflecting the accuracy of the trajectory, also known as the trajectory accuracy). All of them can be selected or some can be selected according to the actual situation. Since the goal is to optimize the trajectory accuracy, the real-time deviation and the maximum deviation of the trajectory are indispensable. Other indicators can be selected according to the needs. If all are selected, taking a four-axis robot as an example, there are a total of 18 state variables for the four axes.

[0080] ;

[0081] Using the trajectory accuracy of the multi-axis robot as the core of the reward formula, appropriate penalty weights are assigned to the performance indicators of each axis that can indirectly affect the overall trajectory accuracy of the robot, and these weights are input into the agent to train the entire environment. The reward formula is not unique; this embodiment provides an example of its establishment. Since the goal is optimal trajectory accuracy, a relatively simple way to establish the reward formula is as follows: If, under manual adjustment, the trajectory accuracy can reach a deviation of 0.7mm, then a high reward can be given to the agent when the maximum trajectory deviation is less than 0.3mm; a small reward can be given when the real-time trajectory deviation is between 0.3mm and 0.7mm; a small penalty can be given when the maximum trajectory deviation is between 0.7mm and 1mm; and a heavier penalty can be given when the maximum trajectory deviation is greater than 1mm, ending the current training round and starting the next round (generally, a large penalty is combined with the round termination condition). Simultaneously, a small penalty is always applied to the real-time trajectory deviation, allowing the agent to learn the goal of minimizing the trajectory at all times.

[0082] ;

[0083] Set termination conditions for each training round and for the overall completion of agent training. The termination condition for each round can be that the maximum trajectory deviation is greater than 1mm. The termination condition for the completion of agent training can be set according to the agent's action update cycle. If the agent's action update frequency is 1000 times per second, the termination condition for the completion of training can be set as an average cumulative reward of 2000 after 20 rounds (or less or more rounds of average reward) (equivalent to the agent receiving a reward of 2 per calculation cycle on average), or the total number of training rounds reaches the upper limit (a maximum of 1000 training rounds can be set).

[0084] Then, the DDPG-based multi-axis robot trajectory control parameter optimization system model is run in the simulation environment, through methods such as... Figure 6 The DDPG workflow diagram shown illustrates the iterative optimization of the policy network until the overall training is completed and the termination condition is met, ultimately yielding a set of PID parameter combinations for each axis that minimizes trajectory error. Figure 6As shown, the DDPG Agent module collects environmental observation state variables, input trajectory target reward formula, and input termination conditions in real time; the DDPG Agent module Action outputs position loop and velocity loop controller parameters in real time to perform closed-loop simulation of the robot's multi-axis motors; the trajectory deviation under the Action output control parameters is calculated in real time based on the closed-loop simulation results; it determines whether the round termination condition is met. If it is met, the current round ends directly, and the next round of training begins; if not, the current round of training continues; the cumulative reward for the round is calculated based on the trajectory target reward formula; it determines whether the overall training completion termination condition is met. If it is met, the training ends, and the final trained controller parameters are output; if not, the next round of training continues.

[0085] Finally, the optimized PID parameters were deployed to the control system of the actual multi-axis robot to execute trajectory tasks, reducing the manual debugging process and improving the accuracy, deployment efficiency and adaptability of robot trajectory control.

[0086] By combining dynamic models with reinforcement learning methods, the motors of each axis of a multi-axis robot can be optimized in a simulation environment. This allows for the rapid acquisition of PID parameters for each axis that meet trajectory accuracy requirements without relying on extensive actual debugging. Furthermore, the gain parameters can be dynamically adjusted according to load changes, thereby significantly improving the trajectory accuracy and synchronization performance of the multi-axis robot under complex motion paths. This ultimately enhances production efficiency and product quality.

[0087] This method aims for optimal trajectory accuracy by adjusting PID parameters across multiple dimensions. Normally, multi-axis robot drive control systems employ three closed-loop control: current loop, velocity loop, and position loop. The current loop, being the inner loop, is generally not adjusted. The velocity loop, using a PI controller, requires adjustment of two parameters: the velocity loop gain Kv and the velocity loop integral time constant Ti. The position loop, using a P controller, requires adjustment of one parameter: the position loop gain Kp. This results in three parameters per axis. A typical four-axis robot requires 12 parameters to adjust. This method eliminates a significant amount of actual debugging and demonstrates superior overall performance compared to manual debugging.

[0088] This application also provides a control device for a multi-axis robot trajectory. It should be noted that the control device for a multi-axis robot trajectory in this application can be used to execute the control method for a multi-axis robot trajectory provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0089] The following describes the control device for the trajectory of a multi-axis robot provided in the embodiments of this application.

[0090] Figure 7 This is a structural block diagram of a control device for a multi-axis robot trajectory according to an embodiment of this application. Figure 7 As shown, the device includes a first construction unit 71, a training unit 72, and a control unit 73. The first construction unit is used to construct a simulation environment including a dynamic model of a multi-axis robot, wherein the dynamic model is used to simulate the rotational inertia and load changes of each axis of the multi-axis robot; the training unit is used to train the controller parameters of each axis using a deep reinforcement learning algorithm based on the simulation environment, thereby obtaining the target controller parameters of each axis after training, wherein the deep reinforcement learning algorithm aims to optimize the overall trajectory accuracy of the multi-axis robot when performing trajectory tasks; the control unit is used to control the trajectory of the multi-axis robot based on the target controller parameters.

[0091] In this embodiment, by constructing a simulation environment with an integrated dynamic model, a deep reinforcement learning algorithm is used to automatically optimize and obtain controller parameters that adapt to changes in rotational inertia and load, with the goal of achieving the best overall trajectory accuracy. This effectively solves the problem of trajectory error accumulation caused by inter-axis coupling and changes in working conditions in multi-axis robots, and improves the trajectory accuracy of multi-axis robots under complex motion paths.

[0092] In some embodiments of this application, the above-mentioned apparatus further includes a second construction unit and a determination unit. The second construction unit is used to construct a deep reinforcement learning model including a policy network and an evaluation network before training the controller parameters of each axis using a deep reinforcement learning algorithm based on the above-mentioned simulation environment. The policy network is used to output controller parameters in the action space according to the state space, and the evaluation network is used to evaluate the action value corresponding to the controller parameters in the action space. The determination unit is used to determine the state space, action space, and reward function of the deep reinforcement learning model. The state space includes at least the real-time trajectory deviation of each axis motor and the maximum deviation of the overall trajectory. The action space includes the controller parameters of each axis, including the position loop gain parameter, speed loop proportional gain parameter, and speed loop integral time constant of each axis. The reward function uses the maximum deviation of the overall trajectory as the optimization index, and the setting of the reward function includes: giving a reward or penalty based on the comparison result of the maximum deviation of the overall trajectory with a preset threshold, and penalizing the real-time trajectory deviation.

[0093] Further, the aforementioned preset thresholds include a first preset threshold, a second preset threshold, and a third preset threshold, and the aforementioned determining unit includes a first processing module, a second processing module, a third processing module, and a fourth processing module. The first processing module is used to grant a first positive reward when the maximum deviation of the overall trajectory is less than or equal to the first preset threshold; the second processing module is used to grant a second positive reward when the maximum deviation of the overall trajectory is greater than the first preset threshold and less than or equal to the second preset threshold; the third processing module is used to grant a first negative penalty when the maximum deviation of the overall trajectory is greater than the second preset threshold and less than or equal to the third preset threshold; and the fourth processing module is used to grant a second negative penalty when the maximum deviation of the overall trajectory is greater than the third preset threshold, wherein the first preset threshold is less than the second preset threshold, and the second preset threshold is less than the third preset threshold.

[0094] By setting preset threshold ranges, a tiered reward and punishment feedback mechanism is constructed, enabling the deep reinforcement learning agent to receive differentiated signal feedback based on the degree of trajectory deviation. High rewards are given when the deviation is extremely small to reinforce the optimal policy, low rewards are given when the deviation is moderate to preserve performance, and penalties of varying degrees are applied when the deviation exceeds a safe range to suppress inferior policies. This mechanism effectively guides the algorithm to converge quickly to a high-precision control region, avoiding training oscillations or slow convergence caused by a single reward signal, thereby improving the efficiency and accuracy of multi-axis robot trajectory control parameter optimization.

[0095] By constructing a deep reinforcement learning model including a policy network and an evaluation network, and defining a state space with real-time trajectory deviation and maximum overall trajectory deviation as core indicators, an action space including three closed-loop key gain parameters, and a reward function based on threshold rewards and penalties, automated and intelligent optimization of PID parameters for each axis of a multi-axis robot is achieved. Leveraging the adaptive characteristics of reinforcement learning, it can automatically balance the control gains of each axis to minimize trajectory errors without requiring manual experience-based tuning, thus improving trajectory accuracy and parameter tuning efficiency under multi-axis linkage control.

[0096] In some embodiments of this application, the state space further includes at least one of the position feedback, position error, speed feedback, and speed error of each axis motor.

[0097] This embodiment enables deep reinforcement learning agents to more keenly capture the coupling effects and dynamic response characteristics (such as overshoot and oscillation tendency) within multi-axis systems. As a result, during the parameter optimization process, it not only optimizes the final trajectory accuracy but also improves the dynamic stability and robustness of the system, accelerates the convergence speed, and enhances the model's generalization ability under complex conditions.

[0098] In some embodiments of this application, the training unit includes an input module, an execution module, an update module, and a determination module. The input module is used to input the state space into the policy network in the simulation environment and output controller parameters in the action space. The execution module is used to execute the control actions corresponding to the controller parameters in the action space and obtain a reward signal based on the execution result. The update module is used to iteratively update the network parameters of the policy network and the evaluation network based on the reward signal until a preset termination condition is met, thereby obtaining the controller parameters output by the policy network. The determination module is used to determine the controller parameters output by the policy network as the target controller parameters.

[0099] By utilizing the iterative mechanism of deep reinforcement learning, the policy network can continuously self-correct based on the real-time control effect (reward signal), automatically explore and lock the optimal combination of PID parameters that maximizes trajectory accuracy, thereby eliminating reliance on human experience and improving the automation level, convergence efficiency and final control accuracy of multi-axis robot trajectory control parameter tuning.

[0100] Furthermore, the aforementioned preset termination conditions include at least one of the following: if the maximum deviation of the overall trajectory in a single training round is greater than a third preset threshold, the training of the single training round is terminated and the next training round begins; if the cumulative number of training rounds reaches a preset number of rounds, or the cumulative reward of an average preset number of rounds reaches a preset reward value, the overall training is terminated.

[0101] This embodiment utilizes a single-round bias threshold mechanism to promptly eliminate inferior strategies, preventing invalid training and accelerating iteration; it prevents training from going on indefinitely by accumulating the number of rounds, controlling computational costs; and it ensures policy stability through average reward convergence judgment, ending training only when the model performance reaches its optimal and stable state. These mechanisms work together to improve the training efficiency, resource utilization, and reliability of the final strategy in deep reinforcement learning for tuning parameters of complex multi-axis robots.

[0102] In some embodiments of this application, the first building unit includes an acquisition module, a configuration module, and an establishment module. The acquisition module acquires a simulation architecture, which includes a dynamics model module and a multi-axis motor control model module for the multi-axis robot. The configuration module configures the dynamics model module to calculate the moment of inertia and load torque of each axis of the multi-axis robot in its current operating state based on a preset trajectory task. The establishment module establishes a data connection between the dynamics model module and the multi-axis motor control model module, so that the moment of inertia and load torque calculated by the dynamics model module are input as input variables to the multi-axis motor control model module to simulate the impact of load changes on trajectory control in the simulation environment.

[0103] By constructing a closed-loop simulation architecture integrating the dynamic model module and the multi-axis motor control model module, the dynamic coupling simulation of load torque and rotational inertia on the motor control system is realized. This enables the deep reinforcement learning agent to be trained in a high-fidelity environment containing real physical disturbances (such as inertia changes and load fluctuations), thereby giving the controller stronger robustness and adaptability, and ensuring that the optimized PID parameters can still maintain high-precision trajectory control performance under actual complex working conditions.

[0104] The aforementioned control device for the multi-axis robot trajectory includes a processor and a memory. The first building unit, training unit, control unit, etc., are all stored as program units in the memory, and the processor executes the program units stored in the memory to achieve the corresponding functions. All of the above modules are located in the same processor; alternatively, the various modules may be located in different processors in any combination.

[0105] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0106] This invention provides a computer-readable storage medium that includes a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform the control method for the multi-axis robot trajectory.

[0107] This invention provides a processor for running a program, wherein the program executes the multi-axis robot trajectory control method during runtime.

[0108] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the multi-axis robot trajectory control method described above. The device described herein can be a server, PC, PAD, mobile phone, etc.

[0109] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform the steps of initializing the control method with the above-described multi-axis robot trajectory.

[0110] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0111] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0116] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0117] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0118] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0120] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for controlling the trajectory of a multi-axis robot, characterized in that, include: A simulation environment is constructed that includes a dynamic model of a multi-axis robot, which is used to simulate the rotational inertia and load changes of each axis of the multi-axis robot; Based on the simulation environment, a deep reinforcement learning algorithm is used to train the controller parameters of each axis to obtain the target controller parameters of each axis after training. The deep reinforcement learning algorithm takes the overall trajectory accuracy of the multi-axis robot when performing trajectory tasks as the optimization objective. The trajectory of the multi-axis robot is controlled based on the target controller parameters.

2. The method according to claim 1, characterized in that, Before training the controller parameters of each axis using a deep reinforcement learning algorithm based on the simulation environment, the method further includes: A deep reinforcement learning model is constructed, comprising a policy network and an evaluation network, wherein the policy network is used to output controller parameters in the action space based on the state space, and the evaluation network is used to evaluate the action value corresponding to the controller parameters in the action space; Determine the state space, action space, and reward function of the deep reinforcement learning model; The state space includes at least the real-time trajectory deviation of each axis motor and the maximum deviation of the overall trajectory; The motion space includes the controller parameters for each axis, which include the position loop gain parameter, the velocity loop proportional gain parameter, and the velocity loop integral time constant for each axis. The reward function uses the maximum deviation of the overall trajectory as the optimization index. The setting of the reward function includes: giving a reward or penalty based on the comparison result of the maximum deviation of the overall trajectory and a preset threshold, and penalizing the real-time deviation of the trajectory.

3. The method according to claim 2, characterized in that, Based on the simulation environment, a deep reinforcement learning algorithm is used to train the controller parameters of each axis to obtain the trained target controller parameters for each axis, including: In the simulation environment, the state space is input to the policy network, and the controller parameters in the action space are output. Execute the control actions corresponding to the controller parameters in the action space, and obtain a reward signal based on the execution result; Based on the reward signal, the network parameters of the policy network and the evaluation network are iteratively updated until a preset termination condition is met, and the controller parameters output by the policy network are obtained. The controller parameters output by the policy network are determined as the target controller parameters.

4. The method according to claim 2, characterized in that, The preset thresholds include a first preset threshold, a second preset threshold, and a third preset threshold. Rewards or penalties are given based on the comparison between the maximum deviation of the overall trajectory and the preset thresholds, including: If the maximum deviation of the overall trajectory is less than or equal to the first preset threshold, a first positive reward is given; If the maximum deviation of the overall trajectory is greater than the first preset threshold and less than or equal to the second preset threshold, a second positive reward is given. If the maximum deviation of the overall trajectory is greater than the second preset threshold and less than or equal to the third preset threshold, a first negative penalty is imposed. If the maximum deviation of the overall trajectory exceeds the third preset threshold, a second negative penalty is imposed. Wherein, the first preset threshold is less than the second preset threshold, and the second preset threshold is less than the third preset threshold.

5. The method according to claim 3, characterized in that, The preset termination condition includes at least one of the following: If the maximum deviation of the overall trajectory in a single training round exceeds a third preset threshold, the training for that single training round is terminated, and the next training round begins. The overall training will terminate when the cumulative number of training rounds reaches the preset number of rounds, or when the cumulative reward for an average preset number of rounds reaches the preset reward value.

6. The method according to claim 1, characterized in that, Construct a simulation environment that includes a multi-axis robot dynamics model, including: Obtain the simulation architecture, which includes the dynamics model module and the multi-axis motor control model module of the multi-axis robot; Configure the dynamic model module so that the dynamic model module calculates the rotational inertia and load torque of each axis of the multi-axis robot in the current operating state according to the preset trajectory task; A data connection is established between the dynamic model module and the multi-axis motor control model module, so that the moment of inertia and the load torque calculated by the dynamic model module are input as input variables to the multi-axis motor control model module, so as to simulate the influence of load changes on trajectory control in the simulation environment.

7. The method according to claim 2, characterized in that, The state space also includes at least one of the position feedback, position error, speed feedback, and speed error of each axis motor.

8. A control device for the trajectory of a multi-axis robot, characterized in that, include: The first building unit is used to build a simulation environment including a dynamic model of a multi-axis robot, wherein the dynamic model is used to simulate the rotational inertia and load changes of each axis of the multi-axis robot; The training unit is used to train the controller parameters of each axis based on the simulation environment using a deep reinforcement learning algorithm to obtain the target controller parameters of each axis after training. The deep reinforcement learning algorithm takes the overall trajectory accuracy of the multi-axis robot when performing trajectory tasks as the optimization objective. The control unit is used to control the trajectory of the multi-axis robot based on the target controller parameters.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform the multi-axis robot trajectory control method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a control method for performing a multi-axis robot trajectory according to any one of claims 1 to 7.

Images