Model-free control method and system for single-link robotic arm under cloud-edge collaboration

Abstract

The present invention discloses a model-free control method and system for a single-link robotic arm under cloud-edge-end collaboration; the method comprises: a terminal collects real-time data of the single-link robotic arm, and sends the real-time data to an edge end; the edge end inputs the preprocessed data into a model-free controller that performs adaptive learning of time-varying parameters based on an online Actor-Critic reinforcement learning network, calculates control instructions for the single-link robotic arm, and the terminal controls the movement of the single-link robotic arm according to the control instructions; at the same time, the edge end sends the preprocessed data to the cloud; the cloud end optimizes and updates the parameters of the model-free controller, and transmits the optimized and updated model-free controller parameters to the model-free controller at the edge end; the above steps are repeated until the control task is completed; the present invention is not only applicable to single-link robotic arms, reduces their modeling difficulty, improves control accuracy, and improves computing efficiency, but also has the application prospect of being extended to other single-input and single-output systems.

Description

Technical Field

[0001] The present invention belongs to the field of robotic arm control, and in particular relates to a model-free control method and system for a single-link robotic arm in cloud-edge collaboration. Background Art

[0002] With the advancement of science and technology and the accelerated development of industrial automation, robotic arms have become an indispensable part of the automation field, undertaking various important tasks such as assembly, handling, welding, etc. Single-link robotic arms are widely used in industrial production, medical rehabilitation and other fields due to their simple structure and low cost. Traditional single-link robotic arm control methods are mainly divided into model-based control and data-driven control. Model-based control methods require the establishment of accurate dynamic models, but complex internal factors and external environments make modeling difficult, which affects its in-depth research and application. Data-driven control directly uses rich input and output data to directly design the controller, which has received widespread attention from scholars.

[0003] Direct model-free adaptive control was first proposed by Hou Zhongsheng and Zhu Yuanming in 2013 in the paper "Controller-dynamic-linearization-based model free adaptive control for discrete-time nonlinear systems." IEEE Transactions on Industrial Informatics, 2013, 9(4):2301-2309. It has now developed into a typical data-driven control method, attracting scholars to explore its potential in combination with neural networks to improve the control performance of single-input single-output controlled systems. With the development of cloud computing, edge computing and direct model-free adaptive control methods, new ideas have been provided for the control of single-link robotic arms; the powerful computing and storage capabilities of cloud computing platforms support complex model training and optimization, and edge computing devices realize real-time data processing and rapid response to control instructions. The combination of direct model-free adaptive control methods and networks with stronger learning capabilities can further improve the performance of model-free adaptive control. Therefore, in order to improve the control performance of the single-link robotic arm and combine the advantages of the cloud-edge, the present invention deeply integrates direct model-free adaptive control and reinforcement learning, and proposes a single-link robotic arm model-free control method and system based on cloud-edge collaboration with stronger learning ability. Summary of the invention

[0004] In order to solve the problems existing in the background technology, the purpose of the present invention is to provide a model-free control method for a single-link robotic arm in cloud-edge collaboration, wherein the cloud-edge includes a cloud, an edge, and a terminal, and includes the following steps:

[0005] Step (1): The terminal collects real-time data of the single-link robotic arm through a sensor signal acquisition device, wherein the real-time data includes a real-time expected joint angle and a real-time operating joint angle, and sends the real-time data to the edge end;

[0006] Step (2): the edge end preprocesses the real-time data, inputs the preprocessed data into a model-free controller that performs adaptive learning of time-varying parameters based on an online Actor-Critic reinforcement learning network, calculates a control instruction for the single-link robotic arm, and sends the control instruction to the terminal, and the terminal controls the movement of the single-link robotic arm according to the control instruction; at the same time, the edge end sends the preprocessed data to the cloud;

[0007] Step (3): The cloud optimizes and updates the parameters of the model-free controller based on the data fed back by the edge, and transmits the optimized and updated model-free controller parameters to the model-free controller at the edge;

[0008] Repeat the above steps (1) to (3) until the control task is completed.

[0009] According to the model-free control method for a single-link robotic arm in cloud-edge collaboration provided by the present invention, in step (2), the edge end preprocesses the real-time data and then inputs the data into a model-free controller that performs adaptive learning of time-varying parameters based on an online Actor-Critic reinforcement learning network to calculate a control instruction for the single-link robotic arm. The preprocessing step includes filtering and noise reduction. The calculation of the model-free controller that performs adaptive learning of time-varying parameters based on an online Actor-Critic reinforcement learning network includes the following steps:

[0010] Step (2.1): k At the sampling time, the mathematical formula of the model-free controller is:

[0011] ;

[0012] in, for k The control instructions of the single-link robot arm at the sampling time, for k The time-varying parameter vector at the sampling instant, for No. l elements, , L is the controller pseudo order, L is a positive integer; for k The trajectory tracking error of the single-link robot at the sampling time, , for k The expected joint angle of the single-link robot at the sampling time, for k The operating joint angle of the single-link robot arm at the sampling time; , ; for k Trajectory tracking error of the single-link manipulator at the sampling time The first-order backward difference of ;

[0013] Step (2.2): Construct a time-varying parameter adaptive learning mechanism based on an online Actor-Critic reinforcement learning network:

[0014] The input of the online Actor-Critic reinforcement learning network includes the k Trajectory tracking error of the single-link manipulator at the sampling time , k First-order backward difference of the trajectory tracking error of a single-link manipulator at the sampling time , k Any one or any combination of the high-order backward differences of the trajectory tracking error of the single-link manipulator at the sampling time; the output of the online Actor-Critic reinforcement learning network includes an Actor output part and a Critic output part, and the Actor output part includes the time-varying parameter vector In , Any one or any combination of, the Critic output part includes a value function.

[0015] According to the model-free control method for a single-link robotic arm in cloud-edge collaboration provided by the present invention, in step (3), the cloud optimizes and updates the parameters of the model-free controller based on the data fed back by the edge, and the optimization and updating process includes the following steps:

[0016] Step (3.1): Construct a real-time dynamic linearization model of the single-link robotic arm:

[0017] ;

[0018] in, k is the sampling time, for The real-time operating joint angle of the single-link robotic arm at each moment, ; for k The control instructions of the single-link robot arm at all times, ; for k Pseudo partial derivatives of the single-link manipulator at the sampling moment;

[0019] The pseudo partial derivative The iterative learning law is:

[0020] ;

[0021] in, is the step size factor, is the penalty factor;

[0022] Step (3.2): In the optimization update process, a one-step forward error is used; based on the real-time dynamic linearization model of the single-link manipulator in step (3.1), the mathematical formula for calculating the one-step forward error is:

[0023] ;

[0024] in, for The trajectory tracking error of the single-link robot arm at time; for k +1 The expected joint angle of the single-link robot at the sampling time;

[0025] In the optimization and updating process, the gradient information of the single-link manipulator is used; based on the real-time dynamic linearization model of the single-link manipulator in step (3.1), the mathematical formula for calculating the gradient information of the single-link manipulator is:

[0026] ;

[0027] in, for against The derivative of

[0028] Step (3.3): During the optimization update process, the control instructions of the single-link robotic arm are used. For the time-varying parameter vector middle , , , The mathematical formula for calculating the partial derivative is:

[0029] If the controller pseudo order ,but

[0030] ;

[0031] If the controller pseudo order ,but

[0032] ;

[0033] Step (3.4): Define the time difference function ,in is the value function, is the discount factor; to minimize the system performance index function As the goal, the weights of the Actor-Critic reinforcement learning network are optimized and updated by using the gradient descent method, wherein the weights include the Actor network weights and the Critic network weights;

[0034] Optimization Update The Actor network weights at the sampling time , , , h is the number of hidden layer nodes of the reinforcement learning network:

[0035] If the controller is pseudo-order ,but

[0036] ;

[0037] If the controller is pseudo-order ,but

[0038] , ;

[0039] in, is the learning rate of the Actor network, Use the mathematical formula for the one-step forward error described in step (3.2); is a value with a mean of 0 and a variance of The normal distribution function of is the hidden layer of the reinforcement learning network i The output of each node;

[0040] Optimization Update The Critic network weights at the sampling time , :

[0041] ;

[0042] in, is the learning rate of the Critic network;

[0043] Step (3.5): Optimize updates The controller time-varying parameter vector at the sampling time is In , , , :

[0044] .

[0045] According to the model-free control method of a single-link robotic arm under cloud-edge collaboration provided by the present invention, the Actor-Critic reinforcement learning network adopts a radial basis function network; the radial basis function network adopts a structure with a single hidden layer, that is, a three-layer network structure consisting of an input layer, a single hidden layer, and an output layer.

[0046] The present invention also provides a model-free control system for a single-link robotic arm in cloud-edge-end collaboration, comprising:

[0047] The terminal comprises a single-link mechanical arm running in real time, a data acquisition module, a first data output module, and a first data input module;

[0048] A data acquisition module, used to collect real-time data of the single-link robotic arm, wherein the real-time data includes a real-time expected joint angle and a real-time operating joint angle;

[0049] A first data output module, used to send the real-time data to an edge end;

[0050] A first data input module, used to receive control instructions sent by the edge end;

[0051] The edge end includes a second data input module, a data preprocessing module, a model-free controller calculation module, and a second data output module;

[0052] A second data input module, used for receiving real-time data of the single-link robotic arm uploaded by a terminal;

[0053] A data preprocessing module, used to preprocess the real-time data and then input it into the model-free controller computing module and send it to the cloud;

[0054] Model-free controller calculation module; used to calculate the control instructions of the single-link manipulator through a model-free controller that performs time-varying parameter adaptive learning based on an online Actor-Critic reinforcement learning network;

[0055] A second data output module, used for sending the control instruction to the terminal;

[0056] The cloud comprises a third data input module, a data storage module, a model-free controller optimization and update module, and a third data output module;

[0057] A third data input module, used to receive the pre-processed real-time data uploaded by the edge end;

[0058] A data storage module, used to store the pre-processed real-time data uploaded by the edge end;

[0059] A model-free controller optimization and updating module, used for optimizing and updating the parameters of the model-free controller;

[0060] The third data output module is used to transmit the optimized and updated model-free controller parameters to the edge end.

[0061] Furthermore, the present invention adopts the following technical solutions:

[0062] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned model-free control method of the single-link robotic arm under cloud-edge collaboration.

[0063] Furthermore, the present invention adopts the following technical solutions:

[0064] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the model-free control method of the single-link robotic arm under cloud-edge collaboration is implemented.

[0065] The beneficial technical effects of the present invention are:

[0066] For single-input single-output controlled systems, existing research has not yet involved combining direct model-free adaptive control with reinforcement learning networks, and combining the advantages of cloud-edge terminals to solve the control algorithm calculation and control command rapid response means. The model-free control method and system of a single-link robotic arm under cloud-edge terminal collaboration provided by the present invention at least include the following beneficial technical effects:

[0067] (1) Reduce the difficulty of modeling: The present invention adopts a data-driven control method, which does not require the establishment of an accurate dynamic model, reduces the difficulty of modeling, and is easier to apply in practice;

[0068] (2) Improve control accuracy: By deeply integrating model-free adaptive control and reinforcement learning, this method can achieve stronger learning capabilities, thereby improving the control accuracy of the single-link robotic arm and meeting higher-precision application requirements;

[0069] (3) Improve computing efficiency: Use the powerful computing power of the cloud computing platform for model training and data analysis, and use edge computing devices to achieve real-time data processing and rapid response to control instructions, effectively improving computing efficiency;

[0070] (4) Broadening the scope of application: The control method and system proposed in the present invention are not only applicable to single-link robotic arms, but can also be extended to other single-input and single-output control systems, and have broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a structural schematic diagram of a single-link robotic arm;

[0072] Figure 2 A schematic flow chart of a model-free control method for a single-link robotic arm in cloud-edge collaboration provided by the present invention;

[0073] Figure 3 A schematic diagram of the connection modules of the model-free control system of the single-link robotic arm under cloud-edge collaboration provided by the present invention. DETAILED DESCRIPTION

[0074] The present invention discloses a model-free control method and system for a single-link robotic arm in cloud-edge collaboration; the terminal collects real-time data of the single-link robotic arm and sends it to the edge; the edge preprocesses the data and inputs it into a model-free controller to calculate control instructions; the terminal controls the movement of the single-link robotic arm according to the control instructions, collects real-time data of the single-link robotic arm and sends it to the edge; the cloud optimizes and updates the parameters of the model-free controller according to the real-time data fed back by the edge, and evaluates the control performance until the control task is completed. The control method and system provided by the present invention are not only applicable to single-link robotic arms, reduce their modeling difficulty, improve control accuracy, and enhance computing efficiency, but also have the application prospect of being extended to other single-input and single-output systems.

[0075] The following is a further clear and complete description of the model-free control method and system of the single-link robotic arm under cloud-edge-end collaboration provided by the present invention in conjunction with the accompanying drawings:

[0076] Figure 1 The structural diagram of the single-link robotic arm is given. Figure 2 A flow chart of a model-free control method for a single-link robotic arm in cloud-edge collaboration provided by the present invention is given; the present invention provides a model-free control method for a single-link robotic arm in cloud-edge collaboration, wherein the cloud-edge includes a cloud, an edge, and a terminal, and includes the following steps:

[0077] Step (1): The terminal collects real-time data of the single-link robotic arm through a sensor signal acquisition device, wherein the real-time data includes a real-time expected joint angle and a real-time operating joint angle, and sends the real-time data to the edge end;

[0078] Step (2): the edge end preprocesses the real-time data, inputs the preprocessed data into a model-free controller that performs adaptive learning of time-varying parameters based on an online Actor-Critic reinforcement learning network, calculates a control instruction for the single-link robotic arm, and sends the control instruction to the terminal, and the terminal controls the movement of the single-link robotic arm according to the control instruction; at the same time, the edge end sends the preprocessed data to the cloud;

[0079] Step (3): The cloud optimizes and updates the parameters of the model-free controller based on the data fed back by the edge, and transmits the optimized and updated model-free controller parameters to the model-free controller at the edge;

[0080] Repeat the above steps (1) to (3) until the control task is completed.

[0081] According to the model-free control method for a single-link robotic arm in cloud-edge collaboration provided by the present invention, in step (2), the edge end preprocesses the real-time data and then inputs the data into a model-free controller that performs adaptive learning of time-varying parameters based on an online Actor-Critic reinforcement learning network to calculate a control instruction for the single-link robotic arm. The preprocessing step includes filtering and noise reduction. The calculation of the model-free controller that performs adaptive learning of time-varying parameters based on an online Actor-Critic reinforcement learning network includes the following steps:

[0082] Step (2.1): k At the sampling time, the mathematical formula of the model-free controller is:

[0083] ;

[0084] in, for k The control instructions of the single-link robot arm at the sampling time, for k The time-varying parameter vector at the sampling instant, for No. l elements, , L is the controller pseudo order, L is a positive integer; for k The trajectory tracking error of the single-link robot at the sampling time, , for k The expected joint angle of the single-link robot at the sampling time, for k The operating joint angle of the single-link robot arm at the sampling time; , ; for k Trajectory tracking error of the single-link manipulator at the sampling time The first-order backward difference of ;

[0085] Step (2.2): Construct a time-varying parameter adaptive learning mechanism based on an online Actor-Critic reinforcement learning network:

[0086] The input of the online Actor-Critic reinforcement learning network includes the k Trajectory tracking error of a single-link manipulator at sampling time , k First-order backward difference of the trajectory tracking error of a single-link manipulator at the sampling time , k Any one or any combination of the high-order backward differences of the trajectory tracking error of the single-link manipulator at the sampling time; the output of the online Actor-Critic reinforcement learning network includes an Actor output part and a Critic output part, and the Actor output part includes the time-varying parameter vector In , Any one or any combination of, the Critic output part includes a value function.

[0087] According to the model-free control method for a single-link robotic arm in cloud-edge collaboration provided by the present invention, in step (3), the cloud optimizes and updates the parameters of the model-free controller based on the data fed back by the edge, and the optimization and updating process includes the following steps:

[0088] Step (3.1): Construct a real-time dynamic linearization model of the single-link robotic arm:

[0089] ;

[0090] in, k is the sampling time, for The real-time operating joint angle of the single-link robotic arm at each moment, ; for k The control instructions of the single-link robot arm at all times, ; for k Pseudo partial derivatives of the single-link manipulator at the sampling moment;

[0091] The pseudo partial derivative The iterative learning law is:

[0092] ;

[0093] in, is the step size factor, is the penalty factor;

[0094] Step (3.2): In the optimization update process, a one-step forward error is used; based on the real-time dynamic linearization model of the single-link manipulator in step (3.1), the mathematical formula for calculating the one-step forward error is:

[0095] ;

[0096] in, for The trajectory tracking error of the single-link manipulator at time t, i.e., the one-step forward error; for k +1 The expected joint angle of the single-link robot at the sampling time;

[0097] In the optimization and updating process, the gradient information of the single-link manipulator is used; based on the real-time dynamic linearization model of the single-link manipulator in step (3.1), the mathematical formula for calculating the gradient information of the single-link manipulator is:

[0098] ;

[0099] in, for against The derivative of , that is, the gradient information;

[0100] Step (3.3): During the optimization update process, the control instructions of the single-link robotic arm are used. For the time-varying parameter vector middle , , , The mathematical formula for calculating the partial derivative is:

[0101] If the controller pseudo order ,but

[0102] ;

[0103] If the controller pseudo order ,but

[0104] ;

[0105] Step (3.4): Define the time difference function ,in is the value function, is the discount factor; to minimize the system performance index function As the goal, the weights of the Actor-Critic reinforcement learning network are optimized and updated by using the gradient descent method, wherein the weights include the Actor network weights and the Critic network weights;

[0106] Optimization Update The Actor network weights at the sampling time , , , h is the number of hidden layer nodes of the reinforcement learning network:

[0107] If the controller is pseudo-order ,but

[0108] ;

[0109] If the controller is pseudo-order ,but

[0110] , ;

[0111] in, is the learning rate of the Actor network, Use the mathematical formula for the one-step forward error described in step (3.2); is a value with a mean of 0 and a variance of The normal distribution function of is the hidden layer of the reinforcement learning network i The output of each node;

[0112] Optimization Update The Critic network weights at the sampling time , :

[0113] ;

[0114] in, is the learning rate of the Critic network;

[0115] Step (3.5): Optimize updates The controller time-varying parameter vector at the sampling time is In , , , :

[0116] .

[0117] According to the model-free control method of a single-link robotic arm under cloud-edge collaboration provided by the present invention, the Actor-Critic reinforcement learning network adopts a radial basis function network; the radial basis function network adopts a structure with a single hidden layer, that is, a three-layer network structure consisting of an input layer, a single hidden layer, and an output layer.

[0118] Figure 3A schematic diagram of the connection modules of the model-free control system of a single-link robotic arm in cloud-edge-end collaboration provided by the present invention is given. The present invention also provides a model-free control system of a single-link robotic arm in cloud-edge-end collaboration, including:

[0119] The terminal comprises a single-link mechanical arm running in real time, a data acquisition module, a first data output module, and a first data input module;

[0120] A data acquisition module, used to collect real-time data of the single-link robotic arm through a sensor signal acquisition device, wherein the real-time data includes a real-time expected joint angle and a real-time operating joint angle;

[0121] A first data output module, used to send the real-time data to an edge end;

[0122] A first data input module, used to receive control instructions sent by the edge end;

[0123] The edge end includes a second data input module, a data preprocessing module, a model-free controller calculation module, and a second data output module;

[0124] A second data input module, used for receiving real-time data of the single-link robotic arm uploaded by a terminal;

[0125] A data preprocessing module, used to preprocess the real-time data and then input it into the model-free controller computing module and send it to the cloud;

[0126] Model-free controller calculation module; used to calculate the control instructions of the single-link manipulator through a model-free controller that performs time-varying parameter adaptive learning based on an online Actor-Critic reinforcement learning network;

[0127] A second data output module, used for sending the control instruction to the terminal;

[0128] The cloud comprises a third data input module, a data storage module, a model-free controller optimization and update module, and a third data output module;

[0129] A third data input module, used to receive the pre-processed real-time data uploaded by the edge end;

[0130] A data storage module, used to store the pre-processed real-time data uploaded by the edge end;

[0131] A model-free controller optimization and updating module, used for optimizing and updating the parameters of the model-free controller;

[0132] The third data output module is used to transmit the optimized and updated model-free controller parameters to the edge end.

[0133] Furthermore, the present invention adopts the following technical solutions:

[0134] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned model-free control method of the single-link robotic arm under cloud-edge collaboration.

[0135] Furthermore, the present invention adopts the following technical solutions:

[0136] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the model-free control method of the single-link robotic arm under cloud-edge collaboration is implemented.

[0137] Through the description of the above embodiments, it can be clearly understood by those skilled in the art that the facilities of the present invention can be implemented by means of software plus the necessary general hardware platform. The embodiments of the present invention can be implemented using an existing processor, or by a dedicated processor used for this purpose or other purposes for an appropriate system, or by a hard-wired system. The embodiments of the present invention also include non-transitory computer-readable storage media, which include machine-readable media for carrying or having machine-executable instructions or data structures stored thereon; such machine-readable media can be any available medium that can be accessed by a general or special-purpose computer or other machine with a processor. For example, such machine-readable media can include RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the required program code in the form of machine-executable instructions or data structures, and can be accessed by a general or special-purpose computer or other machine with a processor. When information is transmitted or provided to a machine via a network or other communication connection (hard-wired, or wireless, or a combination of hard-wired and wireless), the connection is also considered a machine-readable medium.

[0138] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A model-free control method for a single-link robotic arm in cloud-edge collaboration, wherein the cloud-edge includes a cloud, an edge, and a terminal, and is characterized in that: The following steps are involved: Step (1): The terminal collects real-time data of the single-link robotic arm, the real-time data including the real-time expected joint angle and the real-time operating joint angle, and sends the real-time data to the edge end; Step (2): the edge end preprocesses the real-time data, inputs the preprocessed data into a model-free controller that performs adaptive learning of time-varying parameters based on an online Actor-Critic reinforcement learning network, calculates a control instruction for the single-link robotic arm, and sends the control instruction to the terminal, and the terminal controls the movement of the single-link robotic arm according to the control instruction; at the same time, the edge end sends the preprocessed data to the cloud; Step (3): The cloud optimizes and updates the parameters of the model-free controller based on the data fed back by the edge, and transmits the optimized and updated model-free controller parameters to the model-free controller at the edge; Repeat the above steps (1) to (3) until the control task is completed.

2. The model-free control method for a single-link robotic arm under cloud-edge-end collaboration according to claim 1 is characterized in that: In step (2), the edge end inputs the preprocessed data into a model-free controller based on an online Actor-Critic reinforcement learning network for time-varying parameter adaptive learning, and the method for calculating the control instructions of the single-link robotic arm includes the following steps: Step (2.1): k At the sampling time, the mathematical formula of the model-free controller is: ； in, for k The control instructions of the single-link robot arm at the sampling time, for k The time-varying parameter vector at the sampling instant, for No. l elements, , L is the controller pseudo order, L is a positive integer; for k The trajectory tracking error of the single-link robot at the sampling time, , for k The expected joint angle of the single-link robot at the sampling time, for k The operating joint angle of the single-link robot arm at the sampling time; , ; for k Trajectory tracking error of the single-link manipulator at the sampling time The first-order backward difference of ; Step (2.2): Construct a time-varying parameter adaptive learning mechanism based on an online Actor-Critic reinforcement learning network: The input of the online Actor-Critic reinforcement learning network includes the k Trajectory tracking error of the single-link manipulator at the sampling time , k First-order backward difference of the trajectory tracking error of a single-link manipulator at the sampling time , k Any one or any combination of the high-order backward differences of the trajectory tracking error of the single-link manipulator at the sampling moment; the output of the online Actor-Critic reinforcement learning network includes an Actor output part and a Critic output part, and the Actor output part includes the time-varying parameter vector In , Any one or any combination of, the Critic output part includes a value function.

3. The model-free control method for a single-link robotic arm in cloud-edge-end collaboration according to claim 2 is characterized in that: In step (3), the cloud optimizes and updates the parameters of the model-free controller based on the data fed back by the edge. The optimization and update process includes the following steps: Step (3.1): Construct a real-time dynamic linearization model of the single-link robotic arm: ； in, k is the sampling time, for The real-time operating joint angle of the single-link robotic arm at each moment, ; for k The control instructions of the single-link robot arm at all times, ; for k Pseudo partial derivatives of the single-link manipulator at the sampling moment; The pseudo partial derivative The iterative learning law is: ； in, is the step size factor, is the penalty factor; Step (3.2): In the optimization update process, a one-step forward error is used; based on the real-time dynamic linearization model of the single-link manipulator in step (3.1), the mathematical formula for calculating the one-step forward error is: ； in, for The trajectory tracking error of the single-link robot arm at time; for k +1 The expected joint angle of the single-link robot at the sampling time; In the optimization and updating process, the gradient information of the single-link manipulator is used; based on the real-time dynamic linearization model of the single-link manipulator in step (3.1), the mathematical formula for calculating the gradient information of the single-link manipulator is: ； in, for against The derivative of Step (3.3): During the optimization update process, the control instructions of the single-link robotic arm are used. For the time-varying parameter vector middle , , , The mathematical formula for calculating the partial derivative is: If the controller pseudo order ,but ； If the controller pseudo order ,but ， ； Step (3.4): Define the time difference function ,in is the value function, is the discount factor; to minimize the system performance index function As the goal, the weights of the Actor-Critic reinforcement learning network are optimized and updated by using the gradient descent method, wherein the weights include the Actor network weights and the Critic network weights; Optimization Update The Actor network weights at the sampling time , , , h is the number of hidden layer nodes of the reinforcement learning network: If the controller is pseudo-order ,but ； If the controller is pseudo-order ,but ， ； in, is the learning rate of the Actor network, Use the mathematical formula for the one-step forward error described in step (3.2); is a value with a mean of 0 and a variance of The normal distribution function of is the hidden layer of the reinforcement learning network i The output of each node; Optimization Update The Critic network weights at the sampling time , : ； in, is the learning rate of the Critic network; Step (3.5): Optimize updates The controller time-varying parameter vector at the sampling time is In , , , : 。 4. The model-free control method for a single-link robotic arm in cloud-edge-end collaboration according to claim 1 is characterized in that: The Actor-Critic reinforcement learning network adopts a radial basis function network; the radial basis function network adopts a structure with a single hidden layer.

5. The model-free control method for a single-link robotic arm in cloud-edge-end collaboration according to claim 1 is characterized in that: In step (2), the edge end preprocesses the real-time data, including filtering and denoising the real-time data.

6. The model-free control method for a single-link robotic arm in cloud-edge-end collaboration according to claim 1 is characterized in that: In the step (1), the terminal collects real-time data of the single-link robotic arm through a sensor signal acquisition device.

7. A model-free control system for a single-link robotic arm in cloud-edge collaboration, characterized in that: include: The terminal comprises a single-link mechanical arm running in real time, a data acquisition module, a first data output module, and a first data input module; A data acquisition module, used to collect real-time data of the single-link robotic arm, wherein the real-time data includes a real-time expected joint angle and a real-time operating joint angle; A first data output module, used to send the real-time data to an edge end; A first data input module, used to receive control instructions sent by the edge end; The edge end includes a second data input module, a data preprocessing module, a model-free controller calculation module, and a second data output module; A second data input module, used for receiving real-time data of the single-link robotic arm uploaded by a terminal; A data preprocessing module, used to preprocess the real-time data and then input it into the model-free controller computing module and send it to the cloud; Model-free controller calculation module; used to calculate the control instructions of the single-link manipulator through a model-free controller that performs time-varying parameter adaptive learning based on an online Actor-Critic reinforcement learning network; A second data output module, used for sending the control instruction to the terminal; The cloud comprises a third data input module, a data storage module, a model-free controller optimization and update module, and a third data output module; A third data input module, used to receive the pre-processed real-time data uploaded by the edge end; A data storage module, used to store the pre-processed real-time data uploaded by the edge end; A model-free controller optimization and updating module, used for optimizing and updating the parameters of the model-free controller; The third data output module is used to transmit the optimized and updated model-free controller parameters to the edge end.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the model-free control method of a single-link robotic arm under cloud-edge collaboration as described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the model-free control method of the single-link robotic arm under cloud-edge collaboration as described in any one of claims 1 to 6.

Images