Narrow space mechanical arm path planning method based on large language model

Abstract

The invention discloses a narrow space mechanical arm path planning method based on a large language model. According to the method, narrow space path planning is carried out by adopting reinforcement learning, a path multi-scale attribute weighting function is constructed and generated as a reward function of reinforcement learning, and a reward function expression form capable of being directly operated and a reinforcement learning training algorithm matched with the reward function expression form are generated under the guidance of a large language model (LLM). The reinforcement learning training indexes are combined to guide the large language model to optimize the generated reward function, the defects that in traditional reinforcement learning, the reward function is difficult to formulate, and a large amount of experience needs to be provided are overcome, a feasible path can be quickly found, and the generated path is high in quality and directly available; balance of path length, safety and execution precision can be achieved through weight dynamic adjustment, rapid migration from narrow space obstacle avoidance to precise assembly and other different scenes is supported, the path quality and the exploration speed can be considered, and the method is suitable for mechanical arm path planning in the complex and narrow environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arms, in particular to a narrow space mechanical arm path planning method based on a large language model. BACKGROUND

[0002] In the field of human-robot collaborative assembly, the path planning technology of the mechanical arm during assembly has always been the forefront of research, especially the path planning of the mechanical arm in a narrow assembly environment. At the same time, the assembly of the mechanical arm in a narrow space requires a feasible route to be planned quickly and accurately during path planning, and the environment may change during the assembly process. Therefore, the algorithm must have real-time and generalization, and be able to solve various path planning problems in narrow environments and can be calculated in real time. Traditional mechanical arm path planning algorithms include RRT, PRM, A*, reinforcement learning, etc., but there are some problems in the application of the mechanical arm path planning in a narrow environment, as follows.

[0003] The current mechanical arm complex scene path planning technology faces three core bottlenecks:

[0004] (1) Limitations of traditional path planning algorithms: traditional methods such as RRT* and A* have high computational complexity, low exploration efficiency, long calculation time, poor path quality, poor complex environment migration, slow response time, and ignore smoothness and kinematic constraints in narrow assembly spaces.

[0005] (2) Inefficient artificial reward function design: traditional reinforcement learning relies on expert experience to design multi-objective reward functions (such as obstacle avoidance weight and end precision threshold), which has a long debugging cycle. Ordinary dilution reward functions converge slowly, consume a lot of invalid computing resources, and cannot guarantee the smoothness of the generated path and poor obstacle avoidance performance.

[0006] (3) Insufficient adaptability to narrow spaces: fixed reward functions are difficult to respond to sudden disturbances and cannot achieve migration in complex narrow environments. The traditional RRT* algorithm has a low success rate in dynamic environments, and cross-scene migration requires re-modeling, which is time-consuming to adjust parameters. SUMMARY

[0007] Therefore, the present application provides a narrow space mechanical arm path planning method based on a large language model, which can quickly find a high-quality feasible path.

[0008] The narrow space mechanical arm path planning method based on a large language model of the present application uses reinforcement learning to plan the path of the mechanical arm, wherein the reward function of the reinforcement learning is weighted by the multi-scale attributes of the generated path, and the specific expression form of the reward function and the reinforcement learning training algorithm matched therewith are obtained by using a large language model, specifically:

[0009] S1: Build a confined space assembly simulation environment and a reinforcement learning environment, and import them into the large language model. Define the assembly task context and task difficulty in the large language model, and let the large language model generate N reward functions that can be directly run by reinforcement learning, as well as matching training algorithms, for the current confined space assembly environment.

[0010] S2: Bring the generated reward function forms and corresponding training algorithms into reinforcement learning for batch training and evaluate the training effect; if there is an evaluation result that meets the set requirements, execute S4; otherwise, execute S3;

[0011] S3, importing each reward function form and its training effect evaluation results into the large language model, allowing the large language model to optimize the reward function form, generate an optimized reward function form, and execute S2;

[0012] S4, select the reward function form and its training algorithm with the best evaluation results as the reward function form and training algorithm finally adopted by reinforcement learning.

[0013] Preferably, in S1, the confined space assembly simulation environment includes: a three-dimensional model and a kinematic model of the robotic arm, a three-dimensional model of the parts to be assembled, a three-dimensional model of obstacles, and a three-dimensional model of the confined space environment.

[0014] Preferably, the 3D model of the robotic arm and the 3D model of the parts to be assembled are converted into urdf files and then imported into the simulation environment. The placement of various obstacles and parts in the initial state and the initial state of the robotic arm are set, and the parameters of contact mechanics are set to obtain the final simulation environment for assembly in a small space.

[0015] Preferably, in S1, the action space of reinforcement learning is the posture of the end effector of the robot arm or the angle values ​​of each joint of the robot arm; the state space is the current angle values ​​of each joint of the robot arm, the posture of the end effector, the posture of the target position, and the position of the obstacle.

[0016] The optimal multi-scale properties of the generated path include path safety, trajectory smoothness, positioning accuracy, energy consumption, and motion efficiency.

[0017] For better results, path safety is represented by the minimum obstacle distance; motion efficiency is represented by the effective motion ratio; positioning accuracy is represented by the end repeat positioning accuracy; energy consumption is represented by the integral of joint torque; and trajectory smoothness is represented by the root mean square of acceleration.

[0018] Preferably, when S2 evaluates the training effect, the evaluation indicators used include: strategy loss, value loss, maximum reward, progress of task progress, and motion smoothness.

[0019] Preferably, the progress of the task is represented by the maximum distance the robotic arm has advanced or the number of times the robotic arm has reached the target point.

[0020] Beneficial effects:

[0021] The present invention adopts reinforcement learning for confined space path planning, and constructs a multi-scale attribute weighted function of the generated path as the reward function of reinforcement learning. The large language model (LLM) is used to guide the generation of a directly executable reward function expression and a matching reinforcement learning training algorithm. The reinforcement learning training indicators are combined to guide the large language model to optimize the generated reward function. This solves the shortcomings of traditional reinforcement learning, which is difficult to formulate reward functions and requires a lot of experience. It can quickly find feasible paths, generate high-quality paths, and directly use them. It can achieve a balance between path length, safety, and execution accuracy through dynamic adjustment of weights, support rapid migration from different scenarios such as confined space obstacle avoidance to precision assembly, and can take into account both path quality and exploration speed. It is suitable for robot arm path planning in complex and confined environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Flow chart of the method of the present invention.

[0023] Figure 2 The figure is a flow chart of the practical operation of the method of the present invention. DETAILED DESCRIPTION

[0024] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0025] The present invention provides a confined space robot path planning method based on a large language model. First, the confined assembly environment in which the robot arm works is modeled, obstacles in the assembly are three-dimensionally modeled and convexly decomposed, and placed in a simulation environment according to work requirements. In addition, a natural language description of the assembly task and a reinforcement learning environment are defined. Then, the reinforcement learning environment code, the confined space assembly environment simulation image, and the assembly task description are input into the large language model. The large language model is prompted by prompt words to output N reinforcement learning multi-scale reward function codes and training algorithm selections that can be directly executed for the current confined environment. Then, the obtained reward function and the corresponding training algorithm are selected. The training algorithm is trained for K rounds in a confined environment and the training effect is evaluated. If the evaluation result does not meet the set requirements, the reward function code and its training results are input into the large language model, and the large language model is used to optimize the reward function code to generate N reward function codes that are more suitable for the current confined environment. Training, evaluation, and optimization are continued until the evaluation result meets the requirements. Finally, the reward function code with the best evaluation result (that is, the most suitable for the current confined environment) and the corresponding training algorithm are selected as the reward function code and training algorithm finally adopted by reinforcement learning. Large-scale reinforcement learning training is carried out to obtain the planning path of the robotic arm in the confined environment.

[0026] The method flow chart and the practical operation flow chart are as follows: Figure 1 and Figure 2 As shown, the specific steps include:

[0027] Step 1: Modeling the Small Environment

[0028] 1) Definition of task background and task difficulty

[0029] First, each specific robotic arm assembly path planning task has a task background and task difficulty. We need to first clarify the task background and difficulty to assist the large language model in understanding the task scenario and difficulty, and design a reward function that better meets the scenario requirements. For example, consider a scenario where a robotic arm is assembling parts in a confined environment. The difficulty lies in the robotic arm's obstacle avoidance and precise installation of parts in a small space. The task background and difficulty can be described using the following structure:

[0030] Mission Background: Automatic high-precision parts assembly by a robotic arm in a narrow space

[0031] Mission Difficulty: Obstacle avoidance of the robotic arm in a confined space; High-precision assembly of parts (accuracy guaranteed to be within ±0.02mm)

[0032] 2) Construction of simulation environment in a small space

[0033] After determining the task definition, we also need to build the robot arm assembly work scene in the simulation environment, mainly including the robot arm's 3D model and kinematic modeling, the 3D model of the parts to be assembled, and the 3D model of the narrow space and obstacles;

[0034] We first perform three-dimensional modeling of the robotic arm, convert the three-dimensional model of the robotic arm into a urdf file that can be used in the simulation environment, and establish a six-degree-of-freedom robotic arm DH parameter model, defining the angular motion range of each joint of the robotic arm and the workspace of the end effector for subsequent simulation solution verification.

[0035] The 3D model of the parts to be assembled and obstacles within the confined space is then converted into a urdf file. To facilitate collision detection during subsequent training, the 3D models of the parts to be assembled and obstacles are first simplified by performing a convex decomposition (decomposing the unstructured obstacles / parts to be assembled into multiple convex polyhedra) before being converted into a urdf file.

[0036] Finally, import the urdf files of the robotic arm, parts to be assembled, and obstacles into a simulation environment (such as pybullet, mujoco, etc.), set the placement of each obstacle and part in the initial state, and the initial state of the robotic arm, and set the parameters of contact mechanics (such as friction coefficient 0.8, elasticity coefficient 0.2) to obtain the final simulation environment for assembly in a small space.

[0037] 3) Strengthen the construction of learning environment

[0038] After building a simulation environment for the robotic arm's working environment, we also need to build a gymnasium-based reinforcement learning environment (using Python). In reinforcement learning, the most important parts are the state space and action space. Generally speaking, for a six-axis robotic arm, the state space is the angles of the six joints and the coordinates of the end effector, and the action space is the angles of the six joints. At the same time, components such as obtaining the current state, calculating the reward function, and executing the current action are also required. Ultimately, a reinforcement learning environment is built that can directly apply various reinforcement learning algorithms to solve a certain goal in the current environment. This is used for subsequent algorithm implementation and reward function optimization. In short, our reinforcement learning environment needs to include the following parts:

[0039] Action space: generally refers to the change in the angle value of the n joints of the robot arm, or the change in the position of the end effector of the robot arm:

[0040] A=[Δω1,Δω2,...,Δω i ,...Δω n ] or A=[Δx,Δy,Δz,Δα,Δβ,Δγ]

[0041] where Δω i is the angle change of joint i, generally -5°≤Δω i ≤5°; Δx, Δy, Δz, Δα, Δβ, and Δγ are the changes in the three-dimensional position and posture of the end effector. The position and posture of the end effector are converted into the angle of each joint through the inverse kinematics solution of the robot.

[0042] For the design of the action space, if the angle value changes of n joints are used, it is best to design it as a Δω incremental control mode to ensure smooth movement of the robotic arm.

[0043] State space: generally the current angle values ​​of n joints and the end effector posture, as well as the current position of the obstacle (which can be represented by the obstacle distance field SDF). Assume there are m obstacles.

[0044] S=[ω1,ω2,...,ω i ,...,ω n ,D1,D2,...,Dj ,...,D m ,x,y,z,α,β,γ,x′,y′,z′,α′,β′,γ′]

[0045] where ω i represents the current angle of joint i, D j represents the shortest distance between the robot arm and the obstacle j, x, y, z, α, β, γ represent the three-dimensional pose of the end effector, and x′, y′, z′, α′, β′, γ′ represent the pose of the target position. j , we first calculate the obstacle distance field for each position in space, which represents the distance from any point p in space to the obstacle surface The closest distance, positive values ​​represent external space, negative values ​​represent internal space, and its mathematical form is as follows:

[0046]

[0047] We also include SDF in the state space and calculate the closest distance D1, D2, ..., D from the robot arm to each obstacle. j ,...,D m , which can better guide reinforcement learning to perform real-time obstacle avoidance.

[0048] Reward function: the reward given for completing the goal

[0049] Observation function: used to obtain the status or other information at a certain time

[0050] Step function: executes actions and returns new observations, rewards, whether it ends, etc.

[0051] Reset function: reset the environment to its initial state and start a new round

[0052] Step 2: Generate a Reward Function for Reinforcement Learning Using a Large Language Model

[0053] The task description, reinforcement learning environment, and confined space assembly simulation environment image constructed in step 1 are input into the large language model. Combined with the prompt word, the large language model generates N directly usable reinforcement learning reward function Python codes for the current confined space assembly task. At the same time, the large language model also matches the optimal reinforcement learning training algorithm corresponding to each reward function Python code.

[0054] The reward function can be configured based on the properties of the generated path. In a confined assembly environment, we primarily focus on properties such as path safety (represented by the minimum obstacle distance (SDF), trajectory smoothness, positioning accuracy, energy consumption, and motion efficiency. Therefore, the reward function we use is a weighted sum of multi-scale generated path properties. This paper uses a large language model, using prompts to guide the large language model to output a multi-scale weighted reward function that can be directly executed in a reinforcement learning environment.

[0055] We first restrict the role of the large language model, clarify its input and output, and provide some reference properties. For example, path safety can be expressed as minimum obstacle distance, and trajectory smoothness can be expressed as root mean square acceleration. This helps the large language model generate a targeted multi-scale reward function. In this step, we use the following prompt words for the large language model:

[0056] #Role

[0057] You are a dedicated and dedicated expert in generating and optimizing reward functions and matching algorithms for robotic arm path planning in confined spaces using reinforcement learning. With deep expertise and extensive experience, you can more accurately and efficiently generate multi-scale reward functions and matching training algorithms based on user-provided task descriptions, the reinforcement learning environment's Python code, and images of the simulation environment, achieving your goals while significantly reducing training resource consumption. You can also provide detailed, clear, and understandable explanations of the generated content.

[0058] ##Skill

[0059] Skill 1: Generate and interpret reward functions and matching algorithms

[0060] (1) Actively ask the user for details of the robot arm path planning task, including but not limited to task objectives, workspace constraints, etc., and receive the Python code of the reinforcement learning environment and the picture of the simulation environment provided by the user.

[0061] (2) Strictly and automatically check whether the received information is complete and accurate. If there are any missing or doubtful information, ask the user clearly and clearly. Perform a comprehensive and in-depth analysis of the received information to generate a more rationally structured, logically clearer, and directly executable Python reward function code. The function form is:

[0062] def_compute_reward(self,obs): #obs is the state space: 6 joint angles + 3 end coordinates

[0063] #Executable python code for the reward function

[0064] return reward

[0065] (3) Automatically based on the characteristics of the task and common needs, it fully pays attention to attributes such as path safety, trajectory smoothness, positioning accuracy, energy consumption, and motion efficiency (evaluation scales can be flexibly added according to user needs). By optimizing the weight distribution method, it generates a more scientific reward function with multi-scale mixed weights. The final output requires the reward function of each scale, the weight corresponding to each attribute, and the reward function of the final combination. The example is as follows:

[0066]

[0067]

[0068] (4) For each generated reward function, automatically evaluate multiple training algorithms that can be directly used in stable_baseline3, give the name of the training algorithm that has been performance evaluated in stable_baseline3 and has the best adaptability to the reward function, and explain in detail the reasons, advantages, and adaptability of the training algorithm to the reward function.

[0069] (5) Explain in detail how the generated reward function achieves the goal for a given task and reduces training resource consumption. The explanation should be based on the characteristics of the task and the principles of the training algorithm to ensure logical coherence and ease of understanding. If the user has difficulty understanding, proactively provide more examples or simplify the explanation.

[0070] (6) The final output format is as follows:

[0071] Reward function: (Related Python code, including each scale of the reward function and its corresponding weight, as well as the final weighted reward function)

[0072] Reward function explanation: (Detailed explanation of how the reward function works and its advantages, including considerations of each scale of the reward function and the rationality of the weight setting)

[0073] Matching training algorithm: PPO (or other training algorithms that can be used directly in stable_baseline3)

[0074] Reasons and advantages of matching training algorithms: (Explain the reasons for choosing this training algorithm, its advantages, and its synergy with the reward function)

[0075] ##limit:

[0076] -Only answer questions related to the reinforcement learning reward function and matching training algorithm for robot arm path planning, and refuse to answer irrelevant topics.

[0077] - The reward function code must comply with the given format requirements and have a reasonable structure and clear logic.

[0078] -The matching training algorithm name must be directly usable in stable_baseline3 and have been performance evaluated.

[0079] - The explanations provided must be clear, accurate, and understandable, combining the characteristics of the task and the principles of the training algorithm. If the user is not satisfied with the explanation, a clear and accurate explanation must be provided again.

[0080] Step 3: Evaluate the reward function for each generation

[0081] For the generated reward function, we will train the same batch of each reward function in a confined environment in this step, and count the relevant indicators during the training process. Here we can divide the indicators into common indicators and special indicators. Common indicators include policy loss (policy_loss), value loss (value_loss), maximum reward (max_reward), and other indicators that are applicable to all reinforcement learning training. They can also be directly obtained through training logs. Special indicators are indicators defined for confined assembly environments. We can also record them during the training process to indicate the quality of the generated path, such as task completion F (for example, in path planning for a confined cabin, it can be the farthest distance the robot arm has advanced; in obstacle avoidance in a simple environment, it can be the number of times the robot arm has reached the target point), trajectory smoothness, etc. The following are some special indicators that we can use to evaluate the generated path in a confined space:

[0082] (Trajectory smoothness)

[0083] mod = min([collision_checker.compute_distance(q) for q in trajectory]) (minimum obstacle distance)

[0084] (Maximum distance traveled)

[0085] We have built a reinforcement learning environment based on gymnasium in step 1, so we can directly use the algorithm in stable_baseline3. The algorithm in stable_baseline3 can be applied to any standard reinforcement learning environment built on gymnasium. In practice, we can use the log after a period of training as one of the common indicators. The indicators in the log include the number of iterations, KL divergence, clipping ratio, entropy loss, value function loss, explained variance and other key indicators of whether the training is stable. At the same time, during training, a callback function is used to count the task completion F, minimum obstacle distance mod and other indicators in the process, as special evaluation indicators of the reward function generated in step 2. Since we mainly consider obstacle avoidance in a small environment, here we choose mod and F.

[0086] In this process, we want to output each reward function and various indicators after a certain number of training rounds, and then structure and organize them. The following is an example of how the content is structured.

[0087] Reward function: (reward function code)

[0088]

[0089] If there is a reward function that meets the above indicator requirements among the generated reward functions, then execute step 5; otherwise, execute step 4;

[0090] Step 4: Optimize the reward function based on the large language model

[0091] In this step, we use the reward function and its training metrics as input to the large language model. The large language model then optimizes the current reward function based on the training metrics. This optimization includes the functional form and weights of the reward function at each scale, and whether other reward functions can be considered to achieve better obstacle avoidance and faster convergence. Ultimately, the optimized reward function is output as a directly executable function. The following are examples of the prompt words used in this step.

[0092] #Role

[0093] You are a senior expert in the field of reward function optimization for confined space robotic arm path planning in reinforcement learning. You are proficient in the analysis and optimization of various reward functions and training parameter indicators, and are committed to improving the effectiveness and convergence speed of reward functions.

[0094] ##Skill

[0095] Skill 1: Automatically Optimize Reward Function

[0096] (1) When the user provides 5 reward functions and their various parameter indicators after 100,000 rounds of training, you need to automatically and carefully analyze the design and training effect of each reward function.

[0097] The input format example is as follows:

[0098]

[0099]

[0100] Where F represents the degree of task completion, which may be the ratio of the farthest position reached by the robot arm to the closest position to the target position in the entire distance.

[0101] (2) Automatically optimize each reward function based on the analysis results. The optimization content includes the functional form of each scale of the reward function, the weight of each scale, and whether other scales can be considered to achieve better obstacle avoidance effect and faster convergence speed.

[0102] (3) Automatically output 5 optimized reward functions. These functions must be Python codes that can be executed directly.

[0103] ##limit:

[0104] -Only focus on the optimization of the reward function of the robot arm path planning in reinforcement learning, and refuse to answer irrelevant topics.

[0105] -The output of the modified reward function must be directly executable Python code.

[0106] Return to step 3, train the optimized reward function in the same batch under a narrow environment and calculate the training indicators until a reward function that meets the indicator requirements is found, and then proceed to step 5.

[0107] Step 5: Select the optimal reward function for training and generate the final path

[0108] The reward function with the best evaluation effect and the matching training algorithm are selected for the final training to obtain the reinforcement learning path planning model suitable for assembly scenarios in confined environments. The final path of the robotic arm is generated based on this model.

[0109] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for manipulator path planning in a narrow space based on a large language model, which uses reinforcement learning for manipulator path planning, characterized by: The reward function of reinforcement learning is a weighted multi-scale attribute of the generated path. The specific expression of the reward function and the corresponding reinforcement learning training algorithm are obtained using a large language model. Specifically, S1: Build a confined space assembly simulation environment and a reinforcement learning environment, and import them into the large language model. Define the assembly task context and task difficulty in the large language model, and let the large language model generate N reward functions that can be directly run by reinforcement learning, as well as matching training algorithms, for the current confined space assembly environment. S2: Bring the generated reward function forms and corresponding training algorithms into reinforcement learning for batch training and evaluate the training results. If there is an evaluation result that meets the set requirements, execute S4, otherwise execute S3; S3, importing each reward function form and its training effect evaluation results into the large language model, allowing the large language model to optimize the reward function form, generate an optimized reward function form, and execute S2; S4, select the reward function form and its training algorithm with the best evaluation results as the reward function form and training algorithm finally adopted by reinforcement learning.

2. The method according to claim 1, wherein In S1, the narrow space assembly simulation environment includes: a three-dimensional model and a kinematic model of the robot arm, a three-dimensional model of the parts to be assembled, a three-dimensional model of obstacles, and a three-dimensional model of the narrow space environment.

3. The method according to claim 2, wherein The 3D models of the robotic arm and the parts to be assembled are converted into urdf files and imported into the simulation environment. The positions of obstacles and parts in the initial state, as well as the initial state of the robotic arm, are set. The parameters of contact mechanics are also set to obtain the final simulation environment for assembly in a narrow space.

4. The method according to claim 1, wherein In S1, the action space of reinforcement learning is the position of the end effector of the robot arm or the angle value of each joint of the robot arm; the state space is the current angle value of each joint of the robot arm, the position of the end effector, the position of the target position, and the position of the obstacle.

5. The method according to claim 1, wherein The multi-scale properties of the generated path include: path safety, trajectory smoothness, positioning accuracy, energy consumption, and motion efficiency.

6. The method according to claim 5, wherein Path safety is expressed by the minimum obstacle distance; motion efficiency is expressed by the effective motion ratio; positioning accuracy is expressed by the end repeat positioning accuracy; energy consumption is expressed by the integral of joint torque; and trajectory smoothness is expressed by the root mean square of acceleration.

7. The method according to claim 1, 5 or 6, wherein: In S1, the following prompt words are used to guide the large language model to generate N reward function forms and corresponding training algorithms that can be directly run by reinforcement learning: #Role You are a dedicated and professional expert in generating and optimizing reward functions and matching algorithms for robotic arm path planning in confined spaces using reinforcement learning. Possessing profound expertise and extensive experience, you can more accurately and efficiently generate multi-scale reinforcement learning reward functions and matching reinforcement learning training algorithms based on user-provided task descriptions, reinforcement learning environment code, and images of the simulation environment, achieving your goals while significantly reducing training resource consumption. You can also provide detailed, clear, and understandable explanations of the generated content. ##Skill Skill 1: Generate and interpret reward functions and matching algorithms (1) Actively ask the user for details of the robot arm path planning task, including but not limited to the task objectives and workspace constraints, and receive the reinforcement learning environment code and simulation environment pictures provided by the user; (2) Strictly and automatically check whether the received information is complete and accurate. If there are any missing or doubtful information, clearly and explicitly ask the user for clarification; conduct a comprehensive and in-depth analysis of the received information to generate a more rationally structured, logically clear, and directly executable reward function code; (3) Automatically pay attention to the multi-scale attributes of the generated path according to the characteristics of the task and common requirements, and generate a more scientific reward function with multi-attribute scale weighting by optimizing the weight distribution method. The final output requires the reward function of each attribute scale, the weight corresponding to each attribute scale, and the reward function of the final combination; (4) Automatically evaluate a variety of reinforcement learning training algorithms that can be directly used in the reinforcement learning environment, and give the name of the reinforcement learning training algorithm that has been evaluated in the reinforcement learning environment and has the best adaptability to the reward function, and explain in detail the reasons for selecting the training algorithm, its advantages, and its adaptability to the generated reward function; (5) Explain in detail how the generated reward function achieves the goal for a given task and reduces training resource consumption. The explanation should be based on the characteristics of the task and the principles of the algorithm to ensure logical coherence and ease of understanding. If users have difficulty understanding, proactively provide more examples or simplify the explanation; (6) The final output is in the form of: Reward function, including the reward function for each dimension and the corresponding weights and final weighted reward function; Reward function explanation: Detailed explanation of how the reward function works and its advantages, including the consideration of each evaluation dimension and the rationality of the weight setting; Matching training algorithm; Reasons and advantages of matching training algorithm: Explain the reasons for choosing this algorithm, its advantages, and its synergy with the reward function; ##limit: -Only answer questions related to the reinforcement learning reward function and matching algorithm for robot arm path planning, and refuse to answer irrelevant topics; - The reward function code must comply with the given format requirements, and the code structure must be reasonable and logically clear; - The matching training algorithm name must be directly usable in the reinforcement learning environment and have been evaluated for performance; - The explanations provided must be clear, accurate, and easy to understand, combining the characteristics of the task and the principles of the algorithm; If the user is not satisfied with the explanation, a clear and accurate explanation must be provided again.

8. The method according to claim 1, wherein When S2 evaluates the training effect, the evaluation indicators used include: strategy loss, value loss, maximum reward, task progress, and motion smoothness.

9. The method according to claim 8, wherein The progress of the task is expressed by the maximum distance the robotic arm has advanced or the number of times the robotic arm has reached the target point.

10. The method according to claim 1, 7 or 8, wherein: In S3, the following prompt words are used to guide the large language model to optimize the reward function form: #Role You are a senior expert in the field of reward function optimization for confined space robotic arm path planning in reinforcement learning. You are proficient in the analysis and optimization of various reward functions and training parameter indicators, and are committed to improving the effectiveness and convergence speed of reward functions. ##Skill Skill 1: Automatically Optimize Reward Function (1) When the user provides multiple reward functions and their various parameter indicators after L rounds of training, you need to automatically and carefully analyze the design and training effect of each reward function; (2) Automatically optimize each reward function based on the analysis results. The optimization content includes the weight of each scale of the reward function and the representation of the reward function at each scale. Can we consider reward functions of other scales to achieve better obstacle avoidance and faster convergence? (3) Automatically output the modified reward function, which must be directly executable code; ##limit: -Only focus on the optimization of the reward function for robot path planning in reinforcement learning, and refuse to answer irrelevant topics; -The output of the modified reward function must be directly executable code.

Images