Mechanical arm assembly method and system based on average reward reinforcement learning

Abstract

The invention relates to a mechanical arm assembling method and system based on average reward reinforcement learning. The method comprises the steps of collecting current state information in an assembly task execution process, and generating an action block sequence through an imitation learning strategy based on a diffusion model; the current state information and the action block sequence serve as input, a residual reinforcement learning strategy based on an APO algorithm is adopted, real-time fine adjustment is conducted on the action block sequence, and therefore the actual execution action of the mechanical arm is generated; and through closed-loop control, real-time feedback and adjustment, the mechanical arm can complete a long-sight-distance complex assembly task. According to the method, the completion rate and efficiency of the long-sight-distance complex assembly task can be improved.

Description

Technical Field

[0001] This invention relates to a robotic arm assembly method and system based on average reward reinforcement learning, belonging to the field of robotic arm assembly operation control technology. Background Technology

[0002] In complex assembly tasks with long line-of-sight for robotic arms, the arm is required to complete a sequence of actions in multiple stages, such as grasping, adjusting posture, insertion, and fixing. In recent years, imitation learning, or behavior cloning (BC), has been widely applied in robotic arm operations, especially when combined with diffusion policies and action chunking techniques, effectively handling long line-of-sight tasks. For example, existing techniques have proposed a behavior cloning strategy based on diffusion policies, which significantly improves the planning capability for long line-of-sight tasks by predicting action sequences over multiple future time steps. However, this method suffers from performance saturation in high-precision tasks; that is, as the amount of demonstration data increases, the task completion rate cannot be further improved. This is mainly attributed to the lack of closed-loop control during execution, making it unable to adapt to distribution offsets or execution deviations in real time.

[0003] To address the limitations of behavior cloning, researchers have attempted to fine-tune behavior cloning policies using reinforcement learning (RL). For example, existing techniques propose a residual reinforcement learning method (ResidualRL), which adds residual reinforcement learning actions to the base policy outputs to improve task performance. Another approach utilizes a discounted reward-based Proximal Policy Optimization (PPO) algorithm to train residual policies and modify the action sequences generated by behavior cloning, introducing some closed-loop control capabilities. However, existing PPO algorithms rely on a discount factor γ to calculate the expected value of the reward, leading to a tendency for discounted reward-based reinforcement learning algorithms to optimize short-term gains. In long-range assembly tasks, long-term gains cannot be ignored, but discounted reward-based PPO algorithms struggle to effectively capture long-term benefits, limiting their performance in long-range tasks. Furthermore, although Blackwell's optimality theory suggests that when the discount factor γ is close to 1, the optimal policy based on the discount criterion can approximate the long-term average reward, research shows that a discount factor close to 1 increases the algorithm's solution difficulty, leading to non-convergence or instability. In practical applications, mainstream deep reinforcement learning algorithms often experience performance degradation or even fail when γ approaches 1, which further highlights the limitations of reinforcement learning algorithms based on discounted rewards in long-range tasks.

[0004] In summary, existing behavioral cloning strategies lack closed-loop control, making it impossible to further improve the assembly completion rate by increasing the amount of data in high-precision tasks, leading to performance saturation. Furthermore, existing PPO algorithms weight future rewards by discount factors, neglecting long-term benefits, which limits the effectiveness of discount-reward-based reinforcement learning and makes it difficult to optimize long-term reward strategies in long-range tasks. Summary of the Invention

[0005] This invention provides a robotic arm assembly method and system based on average reward reinforcement learning, aiming to solve at least one of the technical problems existing in the prior art.

[0006] The technical solution of this invention relates to a robotic arm assembly method based on average reward reinforcement learning. The method according to this invention includes the following steps:

[0007] S100: Collect the current state information during the assembly task execution process, and generate action block sequences through an imitation learning strategy based on a diffusion model;

[0008] S200. Using the current state information and the action block sequence as input, a residual reinforcement learning strategy based on the APO algorithm is used to fine-tune the action block sequence in real time to generate the actual actions performed by the robotic arm.

[0009] The S300 uses closed-loop control for real-time feedback and adjustment, enabling the robotic arm to complete complex assembly tasks with long line-of-sight.

[0010] Furthermore, step S100 includes: acquiring the state at time t. The input is fed into the imitation learning strategy, and a dimension T is sampled. a ×A b noise matrix Each element of this matrix is ​​independently sampled from a standard normal distribution N(0,1) with a mean of 0 and a variance of 1. The input is fed into the imitation learning strategy, D-step denoising is performed, and a continuous T-step output is generated. a Actions at that moment in, To mimic the state space dimension of the learning strategy, To mimic the action space dimension of the learning strategy, the denoising formula for the k-th step is expressed as follows:

[0011]

[0012] In the formula, k is the number of denoising steps, k = 1, ..., D, v k For the range from v1 = 0.0001 to v D=0.02, a linearly increasing variance scheduling coefficient. To learn by imitation The output noise, θ is the parameter of the imitation learning strategy, u k =1-v k , ∈ k Let T be the dimension obtained by sampling at step k. a ×A b The noise matrix, where each element of the matrix is ​​independently sampled from a standard normal distribution N(0,1) with mean 0 and variance 1, Y t This refers to the noise reduction output action in the final step of the imitation learning strategy.

[0013] Furthermore, in step S200, the action output by the imitation learning strategy is... The action output by the residual reinforcement learning policy based on the APO algorithm The two parts are added together to obtain the actual action performed by the robotic arm, which is represented as follows:

[0014]

[0015] In the formula, ω is a coefficient, 0 < ω < 1. This refers to the actual actions performed by the robotic arm.

[0016] Furthermore, in step S200,

[0017] After obtaining T a Actions at that moment In the middle, select the actions of the first b time steps, 0 <b≤T a The APO algorithm is used to train a residual reinforcement learning policy, which is then fine-tuned for the actions at these b time points. The input state of the residual reinforcement learning policy at these b time points is...

[0018] in For the state space dimension of the residual reinforcement learning strategy, Actions from input to environment It is obtained through interaction with the environment.

[0019] The reward function is set as follows:

[0020]

[0021] The training process goes through K rounds, and in each round, a single data point containing N time steps is collected. The trajectory, and the average value η of the trajectory. new It is represented as follows:

[0022]

[0023] In the formula, ζ is the sliding coefficient, 0 < ζ < 1, η old This is the average value from the previous round; This represents the reward of the residual reinforcement learning strategy at time n;

[0024] Use state value functions Estimate the state at time n The value of Γ is given, where φ is a parameter of the state-value function. The average value Γ of the state-value function over N time points for this trajectory is calculated. new It is represented as follows:

[0025]

[0026] In the formula, Γ old This is the average value of the previous round, where V φ The parameter φ is used with the following mean square error loss function L φ renew:

[0027]

[0028] In the formula, This represents the target state value at time n. Represents the state at time n+1. The value of .

[0029] Furthermore, in step S200,

[0030] Residual reinforcement learning strategy based on APO algorithm The objective function is expressed as follows:

[0031]

[0032] In the formula, For residual reinforcement learning strategies The parameters, This represents the dominance function at time n. δ n+t δ represents the time-series difference error at time n+t. n This represents the time-series difference error at time n. λ is a coefficient, 0 < λ < 1. This represents the state of the reinforcement learning policy based on the input residual at time n. Output Action The strategy from the previous round, Indicates to The cutoff is between [1-ε, 1+ε], where ε is the cutoff factor, and 0<ε<1.

[0033] Furthermore, in step S100, the imitation learning strategy learns and trains from expert demonstration data, generates an initial assembly action sequence based on the state information of the assembly task, and predicts the action sequence for multiple future time steps through action segmentation technology.

[0034] Furthermore, in step S100, the expert demonstration data includes the state sequence and action sequence of the assembly task.

[0035] Furthermore, in step S100, the imitation learning strategy uses a U-net network as the strategy architecture for the diffusion model, and trains the diffusion model using expert demonstration data. The training steps include:

[0036] Randomly select T from the assembly action sequence of the expert demonstration data. a Actions at that moment Add a dimension of T a ×A b noise matrix Each element of this matrix is ​​independently sampled from a standard normal distribution N(0,1) with a mean of 0 and a variance of 1, and is represented as follows:

[0037]

[0038] In the formula, T represents a The action of adding T-step noise at each moment. u T =1-v T v T For the range from v1 = 0.0001 to v D =0.02 The T-th step coefficient selected from the linearly increasing variance scheduling coefficients.

[0039] The expert demonstration data will assemble the state sequence at time t. As conditional information, it is fused into the U-net network, and the noise is predicted and added through the U-net network. Its mean square error loss function L θ It is expressed as follows:

[0040]

[0041] In the formula, To learn by imitation Output noise.

[0042] The present invention also relates to a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described method.

[0043] The present invention also relates to a robotic arm assembly system based on average reward reinforcement learning, the system including a computer device that includes the aforementioned computer-readable storage medium.

[0044] The beneficial effects of this invention are as follows:

[0045] This invention relates to a robotic arm assembly method and system based on average reward reinforcement learning. It integrates imitation learning and average policy optimization (APO) to improve the completion rate and efficiency of complex assembly tasks with long line-of-sight, and is particularly suitable for high-precision assembly scenarios with long line-of-sight. This invention employs the APO algorithm as the residual reinforcement learning strategy and fine-tunes the diffusion-based imitation learning strategy. Specifically targeting the characteristics of complex assembly tasks with long line-of-sight, it significantly improves task completion rate and assembly efficiency by optimizing the long-term average reward.

[0046] Compared to existing PPO algorithms based on discounted rewards, this invention employs an APO algorithm based on average rewards, applying it to residual strategy fine-tuning for robotic arm assembly tasks. This addresses the issue of neglecting long-term gains in long-range tasks. Specifically, this invention optimizes long-term average rewards through the APO algorithm, combined with closed-loop control, significantly improving the completion rate of long-range tasks. Furthermore, the residual reinforcement learning strategy based on average rewards in this invention places greater emphasis on long-term gains, and real-time fine-tuning reduces ineffective actions, shortening assembly time and improving assembly efficiency. Attached Figure Description

[0047] Figure 1 This is a flowchart of the robotic arm assembly strategy training according to the method of the present invention.

[0048] Figure 2 This is a basic flowchart of the method according to the present invention.

[0049] Figure 3 This is a simplified schematic diagram of a general robotic arm structure according to an embodiment of the present invention.

[0050] Figure 4 This is a framework for a robotic arm assembly strategy according to the method of the present invention.

[0051] Figure 5 This is the learning curve of the method according to the present invention in the assembly task. Detailed Implementation

[0052] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with the embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention.

[0053] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. The singular forms "a," "described," and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and not for limiting the invention. The term "and / or" as used herein includes any combination of one or more of the associated listed items.

[0054] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from one another. For example, a first element may also be referred to as a second element without departing from the scope of this disclosure, and similarly, a second element may also be referred to as a first element. Any and all instances or exemplary language (“e.g.,” “such as,” etc.) provided herein are intended only to better illustrate embodiments of the invention and, unless otherwise required, do not impose a limitation on the scope of the invention.

[0055] Reference Figures 1 to 5 In some embodiments of the present invention, the robotic arm assembly method based on average reward reinforcement learning according to the present invention includes at least the following steps:

[0056] S100: Collect the current state information during the assembly task execution process, and generate action block sequences through an imitation learning strategy based on a diffusion model;

[0057] S200. Using the current state information and the action block sequence as input, a residual reinforcement learning strategy based on the APO algorithm is used to fine-tune the action block sequence in real time to generate the actual actions performed by the robotic arm.

[0058] The S300 uses closed-loop control for real-time feedback and adjustment, enabling the robotic arm to complete complex assembly tasks with long line-of-sight.

[0059] This invention addresses the performance saturation problem of behavioral cloning in high-precision tasks and the shortcomings of discounted reward-based reinforcement learning (such as PPO) in optimizing long-term gains for long-range tasks. It proposes a robotic arm assembly method and system based on average reward reinforcement learning to improve the completion rate and assembly efficiency of complex long-range tasks. See also... Figure 1 and Figure 2The illustrated robotic arm assembly strategy framework flow demonstrates how, in actual assembly tasks, this invention combines an imitation learning strategy and a residual strategy based on average reward to generate the robotic arm's execution actions. Through real-time feedback and adjustment, it ensures high precision and efficiency in completing complex assembly tasks with long line-of-sight distances. It should be noted that this invention's method is applicable to any robotic arm with an end effector, such as those described in [reference needed]. Figure 3 The robotic arm includes a base, one or more motion units, such as rotary joints, locating joints or other motion mechanisms, and an end effector, wherein the end effector is capable of performing tasks such as grasping, positioning or assembly.

[0060] Specifically, the present invention provides a robotic arm assembly method and system that integrates imitation learning and average reward-based reinforcement learning (APO). This method improves the completion rate and efficiency of complex assembly tasks at long line-of-sight distances, and is particularly suitable for high-precision assembly scenarios at long line-of-sight distances. The invention employs the APO algorithm as the residual reinforcement learning strategy and fine-tunes the diffusion-based imitation learning strategy. Targeting the characteristics of complex assembly tasks at long line-of-sight distances, it significantly improves task completion rate and assembly efficiency by optimizing long-term average rewards, overcoming the limitation of traditional reinforcement learning algorithms based on discounted rewards (such as PPO) that prioritize short-term gains while neglecting long-term benefits.

[0061] In the generation of the imitation learning strategy in this invention, a diffusion model is used to learn and train from expert demonstration data, generating an initial assembly action sequence based on the state information of the assembly task, supporting planning for long-range tasks. Furthermore, in the residual reinforcement learning based on average reward in this invention, the APO algorithm is used to train the residual strategy, fine-tuning the action sequence generated by the imitation learning to optimize the long-term average reward to adapt to the dynamic requirements of long-range assembly tasks. Moreover, in the closed-loop control and real-time adjustment of this invention, the residual reinforcement learning strategy is used to correct the actions output by the diffusion model-based imitation learning strategy at each time step, achieving closed-loop control and improving the accuracy and robustness of the assembly process.

[0062] It's important to note that reinforcement learning algorithms based on discounted rewards (such as PPO) tend to optimize short-term gains. In contrast, reinforcement learning algorithms based on average rewards prioritize long-term benefits by optimizing long-term average performance. (See [link to relevant documentation]). Figure 5 This invention applies the APO algorithm to the residual strategy fine-tuning of complex assembly tasks with long line of sight of robotic arms. It uses a reinforcement learning algorithm based on average reward, optimizes the long-term average reward through the trust region method, adds average value constraints to solve the value drift problem, and shows better performance than PPO in continuous control tasks.

[0063] In some embodiments of the present invention, the present invention uses a diffusion model as an imitation learning strategy to learn and train from expert demonstration data, generate an initial assembly action sequence based on the state information of the assembly task, and predict the action sequence for multiple future time steps through action segmentation technology to generate a plan suitable for long-range tasks.

[0064] In one application embodiment, the present invention collects H pieces of expert demonstration data, including the state sequence and action sequence of the assembly task. For the state space dimension, For the action space dimension.

[0065] In one application embodiment, the present invention employs a U-net network as the strategy architecture for the diffusion model, and trains the diffusion model using collected expert demonstration data. During the training phase, T is arbitrarily selected from the action sequences assembled from the expert demonstration data. a Actions at that moment Add a dimension of T a ×A b noise matrix Each element of this matrix is ​​independently sampled from a standard normal distribution N(0,1) with a mean of 0 and a variance of 1, and is represented as follows:

[0066]

[0067] In the formula, T represents a The action of adding T-step noise at each moment. u T =1-v T v T For the range from v1 = 0.0001 to v D =0.02 The T-th step coefficient selected from the linearly increasing variance scheduling coefficients.

[0068] The expert demonstration data will assemble the state sequence at time t. As conditional information, it is fused into the U-net network, and the noise is predicted and added through the U-net network. Its mean square error loss function L θ It is expressed as follows:

[0069]

[0070] In the formula, θ represents the imitation learning strategy. The parameters, To learn by imitation Output noise.

[0071] In some embodiments of this invention, the APO algorithm is used to train the residual reinforcement learning policy, and the actions generated by the imitation learning policy are fine-tuned. The APO algorithm aims to optimize the long-term average reward and uses a trust region method to ensure the stability of the policy improvement. Specifically, at time t, the residual reinforcement learning policy uses the state input by the imitation learning policy. Actions acquired through imitation learning strategies For input, the fine-tuning amount of the output action. To achieve adaptation to dynamic environments.

[0072] In one application embodiment, when training the residual reinforcement learning strategy, the present invention needs to fix the imitation learning strategy. Specifically, the state collected at time t is... The input is fed into the imitation learning strategy, and a dimension T is sampled. a ×A b noise matrix Each element of this matrix is ​​independently sampled from a standard normal distribution N(0,1) with a mean of 0 and a variance of 1. The input is fed into the imitation learning strategy, D-step denoising is performed, and a continuous T-step output is generated. a Actions at that moment in, To mimic the state space dimension of the learning strategy, To mimic the action space dimension of the learning strategy, the denoising formula for the k-th step is expressed as follows:

[0073]

[0074] In the formula, k is the number of denoising steps, k = 1, ..., D, v k For the range from v1 = 0.0001 to v D =0.02, a linearly increasing variance scheduling coefficient. To learn by imitation Output noise, u k =1-v k , ∈ k Let T be the dimension obtained by sampling at step k. a ×A b The noise matrix, where each element of the matrix is ​​independently sampled from a standard normal distribution N(0,1) with mean 0 and variance 1, Y t This refers to the noise reduction output action in the final step of the imitation learning strategy.

[0075] After obtaining T a Actions at that moment In the middle, select the actions of the first b time steps, 0 <b≤T a The APO algorithm is used to train a residual reinforcement learning policy, which is then fine-tuned for the actions at these b time points. The input state of the residual reinforcement learning policy at these b time points is...

[0076] in For the state space dimension of the residual reinforcement learning strategy, Actions from input to environment It is obtained through interaction with the environment.

[0077] In some embodiments of the present invention, the state space of the APO algorithm at time t is... The state is the input to the imitation learning strategy. There are also actions that imitate the output of learning strategies. That is This represents the state space dimension of the APO algorithm. See also... Figure 4 The action output by the imitation learning strategy The action output by the residual reinforcement learning policy based on the APO algorithm The two parts are added together to obtain the actual action performed by the robotic arm, which is represented as follows:

[0078]

[0079] In the formula, ω is a coefficient, 0 < ω < 1. This refers to the actual actions performed by the robotic arm.

[0080] Furthermore, the reward function is set as follows:

[0081]

[0082] The training process goes through K rounds, and in each round, a single data point containing N time steps is collected. The trajectory, and the average value η of the trajectory. new It is represented as follows:

[0083]

[0084] In the formula, ζ is the sliding coefficient, 0 < ζ < 1, η old This is the average value from the previous round; This represents the reward of the residual reinforcement learning strategy at time n;

[0085] Use state value functions Estimate the state at time n The value of Γ is given by φ, where φ is a parameter of the state-value function. The average value Γ of the state-value function over N time points for this trajectory is calculated.new It is represented as follows:

[0086]

[0087] In the formula, Γ old This is the average value of the previous round, where V φ The parameter φ is used with the following mean square error loss function L φ renew:

[0088]

[0089] In the formula, This represents the target state value at time n. Represents the state at time n+1. The value of .

[0090] Furthermore, a residual reinforcement learning strategy based on the APO algorithm. The objective function is expressed as follows:

[0091]

[0092] In the formula, For residual reinforcement learning strategies The parameters, This represents the dominance function at time n. δ n+t δ represents the time-series difference error at time n+t. n This represents the time-series difference error at time n. λ is a coefficient, 0 < λ < 1. This represents the state of the reinforcement learning policy based on the input residual at time n. Output Action The strategy from the previous round, Indicates to The cutoff is between [1-ε, 1+ε], where ε is the cutoff factor, and 0<ε<1.

[0093] Further, see Figure 4 We optimize the APO algorithm based on average reward by using the policy gradient update method to learn the optimal policy for long-range tasks.

[0094] It should be understood that the method steps in the embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The method can use standard programming techniques. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).

[0095] Furthermore, the procedures described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The procedures described herein (or variations and / or combinations thereof) may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program comprises a plurality of instructions executable by one or more processors.

[0096] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices, etc. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RSM, ROM, etc., such that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques described in the invention, the invention may also include the computer itself.

[0097] A computer program can be applied to input data to perform the functions described herein, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including specific visual depictions of physical and tangible objects generated on the display.

[0098] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention, as long as they achieve the technical effects of the present invention by the same means, should be included within the scope of protection of the present invention. Within the scope of protection of the present invention, the technical solutions and / or implementation methods can have various modifications and variations.

Claims

1. A robotic arm assembly method based on average reward reinforcement learning, characterized in that, The method includes the following steps: S100: Collect the current state information during the assembly task execution process, and generate action block sequences through an imitation learning strategy based on a diffusion model; S200. Using the current state information and the action block sequence as input, a residual reinforcement learning strategy based on the APO algorithm is used to fine-tune the action block sequence in real time to generate the actual actions performed by the robotic arm. The S300 uses closed-loop control for real-time feedback and adjustment, enabling the robotic arm to complete complex assembly tasks with long line-of-sight.

2. The method according to claim 1, characterized in that, Step S100 includes: The state at time t is collected The input is fed into the imitation learning strategy, and a dimension T is sampled. a ×A b noise matrix Each element of this matrix is ​​independently sampled from a standard normal distribution N(0,1) with a mean of 0 and a variance of 1. The input is fed into the imitation learning strategy, D-step denoising is performed, and a continuous T-step output is generated. a Actions at that moment in, To mimic the state space dimension of the learning strategy, To mimic the action space dimension of the learning strategy, the denoising formula for the k-th step is expressed as follows: In the formula, k is the number of denoising steps, k = 1, ..., D, v k For the range from v1 = 0.0001 to v D =0.02, a linearly increasing variance scheduling coefficient. To learn by imitation The output noise, θ is the parameter of the imitation learning strategy, u k =1-v k , ∈ k Let T be the dimension obtained by sampling at step k. a ×A b The noise matrix, where each element of the matrix is ​​independently sampled from a standard normal distribution N(0,1) with mean 0 and variance 1, Y t This refers to the noise reduction output action in the final step of the imitation learning strategy.

3. The method according to claim 2, characterized in that, In step S200 The action output by the imitation learning strategy The action output by the residual reinforcement learning policy based on the APO algorithm The two parts are added together to obtain the actual action performed by the robotic arm, which is represented as follows: In the formula, ω is a coefficient, 0 < ω < 1. This refers to the actual actions performed by the robotic arm.

4. The method according to claim 2, characterized in that, In step S200 After obtaining T a Actions at that moment In the middle, select the actions of the first b time steps, 0 <b≤T a The APO algorithm is used to train a residual reinforcement learning policy, which is then fine-tuned for the actions at these b time points. The input state of the residual reinforcement learning policy at these b time points is... in For the state space dimension of the residual reinforcement learning strategy, Actions from input to environment It is obtained through interaction with the environment. The reward function is set as follows: The training process goes through K rounds, and in each round, a single data point containing N time steps is collected. The trajectory, and the average value η of the trajectory. new It is represented as follows: In the formula, ζ is the sliding coefficient, 0 < ζ < 1, η old This is the average value from the previous round; This represents the reward of the residual reinforcement learning strategy at time n; Use state value functions Estimate the state at time n The value of Γ is given, where φ is a parameter of the state-value function. The average value Γ of the state-value function over N time points for this trajectory is calculated. new It is represented as follows: In the formula, Γ old This is the average value of the previous round, where V φ The parameter φ is used with the following mean square error loss function L φ renew: In the formula, This represents the target state value at time n. Represents the state at time n+1. The value of .

5. The method according to claim 4, characterized in that, In step S200 Residual reinforcement learning strategy based on APO algorithm The objective function is expressed as follows: In the formula, For residual reinforcement learning strategies The parameters, This represents the dominance function at time n. δ n+t δ represents the time series difference error at time n+t. n This represents the time-series difference error at time n. λ is a coefficient, 0 < λ < This represents the state of the reinforcement learning policy based on the input residual at time n. Output Action The strategy from the previous round, Indicates to The cutoff is between [1-ε, 1+ε], where ε is the cutoff factor, and 0<ε<1.

6. The method according to claim 1, characterized in that, In step S100, The imitation learning strategy learns and trains from expert demonstration data, generates an initial assembly action sequence based on the state information of the assembly task, and predicts the action sequence for multiple future time steps through action segmentation technology.

7. The method according to claim 6, characterized in that, In step S100, The expert demonstration data includes the state sequence and action sequence of the assembly task.

8. The method according to claim 2, characterized in that, In step S100, The imitation learning strategy uses the U-net network as the strategy architecture for the diffusion model, and trains the diffusion model using expert demonstration data. The training steps include: Randomly select T from the assembly action sequence of the expert demonstration data. a Actions at that moment Add a dimension of T a ×A b noise matrix Each element of this matrix is ​​independently sampled from a standard normal distribution N(0,1) with a mean of 0 and a variance of 1, and is represented as follows: In the formula, T represents a The action of adding T-step noise at each moment. u T =1-v T v T For the range from v1 = 0.0001 to v D =0.02 The T-th step coefficient selected from the linearly increasing variance scheduling coefficients. The expert demonstration data will assemble the state sequence at time t. As conditional information, it is fused into the U-net network, and the noise is predicted and added through the U-net network. Its mean square error loss function L θ It is expressed as follows: In the formula, To learn by imitation Output noise.

9. A computer-readable storage medium having stored thereon program instructions that, when executed by a processor, perform the method as described in any one of claims 1 to 8.

10. A robotic arm assembly system based on average reward reinforcement learning, characterized in that, include: A computer device, the computer device comprising the computer-readable storage medium according to claim 9.

Images