Mechanical arm trajectory planning method and device based on reinforcement learning, equipment and medium

Abstract

The invention relates to a mechanical arm trajectory planning method and device based on reinforcement learning, equipment and a medium. The method comprises the following steps: acquiring a real-time pose parameter, a target pose parameter and a state transition parameter of the mechanical arm; on the basis of the real-time pose parameter, the state transition parameter and the target pose parameter, transferable trajectory information is identified from a historical trajectory database; environment obstacle distribution label information of the mechanical arm is obtained, and a trajectory planning reinforcement learning model of the mechanical arm is selected based on the environment obstacle distribution label information; and the transferable trajectory information is input into a trajectory planning reinforcement learning model matched with the environment obstacle distribution label information, and real-time trajectory information of the mechanical arm is generated. By the adoption of the method, the defects of traditional trajectory planning in dynamic environment adaptability, calculation efficiency and generalization ability can be overcome by fusing real-time sensing, historical data migration and dynamic model matching, and efficient and safe operation of the mechanical arm in an unstructured scene can be achieved.

Description

Technical Field

[0001] This application belongs to the field of reinforcement learning-based robotic arm trajectory planning, and in particular relates to a reinforcement learning-based robotic arm trajectory planning method, device, equipment and medium. Background Technology

[0002] With the development of industrial automation and intelligent manufacturing technologies, robotic arm trajectory planning has become a core research direction in the field of robotics. Early trajectory planning relied on traditional techniques such as geometric path planning, artificial potential field methods, and mathematical optimization methods. These methods generate the motion path of the robotic arm by constructing an environment model, defining obstacle avoidance rules, or solving constrained optimization problems.

[0003] In traditional technologies, robotic arm trajectory planning primarily relies on pre-programmed motion sequences or mathematical methods based on precise kinematic and dynamic models. Model-based methods solve for optimal trajectories through numerical computation, providing theoretically accurate trajectory planning, but they suffer from significant limitations in practical applications. First, existing technologies have low computational efficiency, requiring repeated sampling and searching in complex high-dimensional spaces, resulting in lengthy processing times. Second, existing technologies exhibit poor dynamic adaptability, struggling to adjust the path in real-time when faced with moving obstacles or sudden environmental changes. Furthermore, artificial potential field methods are prone to getting trapped in local minima, leading to trajectory planning failures, while mathematical optimization methods rely on precise modeling to solve complex non-convex problems, limiting their applicability. Summary of the Invention

[0004] Therefore, it is necessary to provide a reinforcement learning-based robotic arm trajectory planning method, device, equipment, and medium that can meet the requirements of efficiency, adaptability, and intelligence for robotic arm trajectory planning in modern industry and complex application scenarios, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a reinforcement learning-based method for robotic arm trajectory planning, including:

[0006] Acquire the real-time pose parameters of the robotic arm, the target pose parameters, and the state transition parameters;

[0007] Based on real-time pose parameters, state transition parameters, and target pose parameters, transferable trajectory information is identified from the historical trajectory database.

[0008] Obtain the environmental obstacle distribution label information of the robotic arm, and select the trajectory planning reinforcement learning model of the robotic arm based on the environmental obstacle distribution label information;

[0009] The transferable trajectory information is input into a trajectory planning reinforcement learning model that matches the distribution label information of environmental obstacles to generate the real-time trajectory information of the robotic arm.

[0010] In one embodiment, transferable trajectory information is identified from a historical trajectory database based on real-time pose parameters, state transition parameters, and target pose parameters, including:

[0011] Based on the state transition parameters, the approximate historical trajectories are selected from the historical trajectory database, and the trajectory control vectors of the approximate historical trajectories are obtained.

[0012] Construct the target trajectory control vector of the robotic arm based on real-time pose parameters and target pose parameters;

[0013] Calculate the error distance between the target trajectory control vector of the robotic arm and the trajectory control vector of the approximate historical trajectory, and generate trajectory difference parameters;

[0014] Based on the trajectory difference parameter, the transferable trajectory is identified from the state approximation historical trajectory, and the trajectory information of the transferable trajectory is set as the transferable trajectory information.

[0015] In one embodiment, the process of identifying a transferable trajectory from a state-approximate historical trajectory based on a trajectory difference parameter includes:

[0016] Identify historical trajectories with trajectory difference parameters greater than a preset trajectory similarity threshold as pre-selected transferable trajectories;

[0017] Set the pre-selected transferable trajectory with the largest trajectory difference parameter as the transferable trajectory.

[0018] In one embodiment, the expression for the trajectory difference parameter is:

[0019]

[0020] In the formula, P TS For trajectory similarity parameters, The target trajectory control vector, For trajectory control vector, and T represents the real-time pose matrix corresponding to the real-time pose parameters of the robotic arm and the target pose matrix corresponding to the target pose parameters of the robotic arm, respectively. TTC These are the trajectory time parameters of the robotic arm. and T represents the real-time pose matrix of the approximate historical trajectory and the target pose matrix of the approximate historical trajectory, respectively. TC The trajectory time parameter is the approximate historical trajectory.

[0021] In one embodiment, the state transition parameters include structural state parameters and performance degradation parameters. Based on the state transition parameters, approximate historical trajectories are selected from a historical trajectory database, including:

[0022] The degree of difference in state transition parameters of historical trajectories in the historical trajectory database is calculated based on structural state parameters and performance degradation parameters.

[0023] Historical trajectories with state transition parameter differences less than a preset state transition parameter difference threshold are selected as approximate historical trajectories.

[0024] In one embodiment, the environmental obstacle distribution label information includes sparse obstacle distribution labels and dense obstacle distribution labels; the trajectory planning reinforcement learning model includes a sparse trajectory planning reinforcement learning model and a dense trajectory planning reinforcement learning model; the objective function of both the sparse and dense trajectory planning reinforcement learning models includes a trajectory obstacle avoidance objective function; the weight of the trajectory obstacle avoidance objective function of the sparse trajectory planning reinforcement learning model is lower than the weight of the trajectory obstacle avoidance objective function of the dense trajectory planning reinforcement learning model; and the selection of the robotic arm's trajectory planning reinforcement learning model based on the environmental obstacle distribution label information includes:

[0025] If the environmental obstacle distribution label information is sparse obstacle distribution label, configure the robot arm's trajectory planning reinforcement learning model as a sparse trajectory planning reinforcement learning model.

[0026] If the environmental obstacle distribution label information is a dense obstacle distribution label, the trajectory planning reinforcement learning model of the robotic arm will be configured as a dense trajectory planning reinforcement learning model.

[0027] In one embodiment, the reinforcement learning-based robotic arm trajectory planning method further includes:

[0028] The transferable trajectory information of the robotic arm with sparse obstacle distribution labels is input into the dense trajectory planning reinforcement learning model to generate the obstacle avoidance trajectory information of the robotic arm.

[0029] In response to the environmental obstacle distribution label information, the robotic arm with sparse obstacle distribution labels identifies obstacles in the real-time trajectory corresponding to the real-time trajectory information during its movement under the control of real-time trajectory information, and updates the real-time trajectory information based on the obstacle avoidance trajectory information.

[0030] Secondly, this application also provides a reinforcement learning-based robotic arm trajectory planning device, comprising:

[0031] The motion parameter acquisition module is used to acquire the real-time pose parameters, target pose parameters, and state transition parameters of the robotic arm;

[0032] The transferable trajectory recognition module is used to identify transferable trajectory information from the historical trajectory database based on real-time pose parameters, state transition parameters, and target pose parameters.

[0033] The environmental parameter acquisition module is used to acquire the environmental obstacle distribution label information of the robotic arm, and select the trajectory planning reinforcement learning model of the robotic arm based on the environmental obstacle distribution label information.

[0034] The real-time trajectory generation module is used to input transferable trajectory information into a trajectory planning reinforcement learning model that matches the distribution label information of environmental obstacles, thereby generating real-time trajectory information of the robotic arm.

[0035] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method as described in any of the first aspects of this application.

[0036] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the first aspects of this application.

[0037] The aforementioned reinforcement learning-based robotic arm trajectory planning method, device, equipment, and medium, by acquiring the robotic arm's pose parameters, target pose parameters, and state transition parameters in real time, can dynamically adjust the planning logic based on the current environmental state, responding in real time to changes in obstacle distribution and target pose adjustment. By utilizing historical trajectory databases to identify transferable trajectory information, past successful experiences are transformed into prior knowledge, effectively reducing the sample dependence of reinforcement learning, decreasing the number of trial and error attempts for the robotic arm in new tasks, and shortening the training cycle. Furthermore, the introduction of state transition parameters enhances the model's generalization ability across different task scenarios, avoiding the inefficiency of learning from scratch. Dynamically selecting an appropriate reinforcement learning model based on environmental obstacle distribution label information significantly improves the multi-objective optimization capability of trajectory planning. Using transferable trajectory information as input conditions for the reinforcement learning model reduces online computation. Additionally, the model-environment label matching mechanism avoids parameter redundancy. Compared to traditional high-dimensional space optimization algorithms, this method significantly improves trajectory planning efficiency while maintaining trajectory accuracy. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1A schematic diagram illustrating the application environment of a reinforcement learning-based robotic arm trajectory planning method provided in one embodiment of this application;

[0040] Figure 2 A flowchart illustrating a reinforcement learning-based robotic arm trajectory planning method provided in one embodiment of this application;

[0041] Figure 3 A flowchart illustrating a method for identifying transferable trajectory information according to an embodiment of this application;

[0042] Figure 4 A flowchart illustrating another reinforcement learning-based robotic arm trajectory planning method provided in one embodiment of this application;

[0043] Figure 5 This is a schematic diagram of a device for a reinforcement learning-based robotic arm trajectory planning method provided in one embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] The reinforcement learning-based robotic arm trajectory planning method provided in this application can be applied to, for example... Figure 1 In the application environment shown, sensor terminal 102 and drive terminal 103 can communicate with server 101 through a communication channel. Database 104 can store the data that server 101 needs to process. Database 104 can be integrated into server 101 or placed in the cloud or other network servers. The server can obtain the current pose and desired pose of the robotic arm through sensor terminal 102, and generate a control strategy for the robotic arm by combining the basic parameter data of the robotic arm in database 104. Server 101 can control drive terminal 103 to achieve motion control of the robotic arm based on the generated control strategy. Server 101 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0046] In one exemplary embodiment, such as Figure 2 As shown, a reinforcement learning-based robotic arm trajectory planning method is provided, which is then applied to... Figure 1 Taking server 101 as an example, the explanation includes the following steps S201 to S204. Wherein:

[0047] Step S201: Obtain the real-time pose parameters, target pose parameters, and state transition parameters of the robotic arm.

[0048] Specifically, server 101 can obtain the real-time pose parameters of the robotic arm through the sensor terminal, obtain the state transition parameters of the robotic arm through the database, receive the control target information of the robotic arm, and extract the target pose parameters of the robotic arm from the control target information.

[0049] Step S202: Based on real-time pose parameters, state transition parameters and target pose parameters, transferable trajectory information is identified from the historical trajectory database.

[0050] Specifically, server 101 can identify transferable trajectory information from the constructed historical trajectory database by acquiring the real-time pose parameters, state transition parameters, and target pose parameters of the robotic arm.

[0051] Optionally, the server can calculate the degree of difference between the target trajectory and the historical trajectory in the historical trajectory database based on the real-time pose parameters, state transition parameters and target pose parameters of the acquired robotic arm, select the historical trajectory with the smallest degree of difference as the transferable trajectory, and identify the trajectory information of the transferable trajectory as the transferable trajectory information.

[0052] Step S203: Obtain the environmental obstacle distribution label information of the robotic arm, and select the trajectory planning reinforcement learning model of the robotic arm based on the environmental obstacle distribution label information.

[0053] Schematic, the environmental obstacle distribution label information of the robotic arm can be used to characterize the density of obstacles in the robotic arm's working environment.

[0054] Step S204: Input the transferable trajectory information into the trajectory planning reinforcement learning model that matches the environmental obstacle distribution label information to generate the real-time trajectory information of the robotic arm.

[0055] Optionally, the trajectory planning reinforcement learning model that matches the environmental obstacle distribution label information can be a trajectory planning reinforcement learning model trained using samples with the same density of obstacles in the working environment of the robotic arm that correspond to the environmental obstacle distribution label information and an objective function.

[0056] In the aforementioned reinforcement learning-based robotic arm trajectory planning method, transferable trajectory information can be identified from a historical trajectory database, enabling the rapid acquisition of an initial trajectory similar to the current task, thereby improving planning efficiency in complex environments. By inputting the transferable trajectory information into a trajectory planning reinforcement learning model that matches environmental obstacle distribution label information, adaptive trajectory planning for the robotic arm in different environments can be achieved, effectively addressing dynamically changing work scenarios and improving the intelligence and accuracy of trajectory planning. By acquiring the environmental obstacle distribution label information of the robotic arm and selecting a suitable trajectory planning reinforcement learning model based on this information, the robotic arm can quickly identify the characteristics of the current environment and select the optimal planning model, improving the versatility and flexibility of the robotic arm in different scenarios.

[0057] In one optional embodiment of this application, such as Figure 3 As shown, transferable trajectory information is identified from the historical trajectory database based on real-time pose parameters, state transition parameters, and target pose parameters, including:

[0058] Step S301: Based on the state transition parameters, filter out the approximate historical trajectory from the historical trajectory database and obtain the trajectory control vector of the approximate historical trajectory.

[0059] Step S302: Construct the target trajectory control vector of the robotic arm based on the real-time pose parameters and the target pose parameters.

[0060] Step S303: Calculate the error distance between the target trajectory control vector of the robotic arm and the trajectory control vector of the historical trajectory that is similar to the state, and generate trajectory difference parameters.

[0061] Step S304: Identify the transferable trajectory from the approximate historical trajectory based on the trajectory difference parameter, and set the trajectory information of the transferable trajectory as transferable trajectory information.

[0062] In the aforementioned reinforcement learning-based robotic arm trajectory planning method, by selecting approximate historical trajectories from the historical trajectory database and calculating trajectory difference parameters to identify transferable trajectories, it is possible to accurately match the historical trajectory with the highest similarity to the current task, greatly shortening the preparation time in the early stage of trajectory planning and improving the efficiency of the robotic arm task initiation phase. By constructing the target trajectory control vector of the robotic arm based on real-time pose parameters and target pose parameters, and calculating the error distance with the trajectory control vector of the historical trajectory, the similarity between the historical trajectory and the current task can be quantified, significantly improving the trajectory matching accuracy. By identifying transferable trajectories from approximate historical trajectories based on trajectory difference parameters, it is possible to ensure that the selected transferable trajectory is highly consistent with the current task in terms of state and control logic, achieving efficient reuse of historical data.

[0063] In an optional embodiment of this application, identifying a transferable trajectory from approximate historical trajectories based on a trajectory difference parameter may include:

[0064] Identify historical trajectories whose trajectory difference parameter is greater than the preset trajectory similarity threshold as pre-selected transferable trajectories.

[0065] Set the pre-selected transferable trajectory with the largest trajectory difference parameter as the transferable trajectory.

[0066] In an optional embodiment of this application, the expression for the trajectory difference parameter can be:

[0067]

[0068] In the formula, P TS For trajectory similarity parameters, The target trajectory control vector, For trajectory control vector, and T represents the real-time pose matrix corresponding to the real-time pose parameters of the robotic arm and the target pose matrix corresponding to the target pose parameters of the robotic arm, respectively. TTC These are the trajectory time parameters of the robotic arm. and T represents the real-time pose matrix of the approximate historical trajectory and the target pose matrix of the approximate historical trajectory, respectively. TC The trajectory time parameter is the approximate historical trajectory.

[0069] Schematic representation of the target pose matrix based on the target pose parameters of the robotic arm. Subtract the real-time pose matrix corresponding to the real-time pose parameters of the robotic arm Then divide by the trajectory time parameter T of the robotic arm TTC The obtained target trajectory control vector The amount It can be used to measure the movement rate of a robotic arm.

[0070] In an optional embodiment of this application, the state transition parameters include structural state parameters and performance degradation parameters. Selecting approximate historical trajectories from a historical trajectory database based on the state transition parameters may include:

[0071] The state transition parameter difference of historical trajectories in the historical trajectory database is calculated based on structural state parameters and performance degradation parameters.

[0072] Historical trajectories with state transition parameter differences less than a preset state transition parameter difference threshold are selected as approximate historical trajectories.

[0073] In an optional embodiment of this application, the environmental obstacle distribution label information includes sparse obstacle distribution labels and dense obstacle distribution labels; the trajectory planning reinforcement learning model includes a sparse trajectory planning reinforcement learning model and a dense trajectory planning reinforcement learning model; the objective function of both the sparse and dense trajectory planning reinforcement learning models includes a trajectory obstacle avoidance objective function; the weight of the trajectory obstacle avoidance objective function in the sparse trajectory planning reinforcement learning model is lower than the weight of the trajectory obstacle avoidance objective function in the dense trajectory planning reinforcement learning model. Please refer to [reference needed]. Figure 4 A reinforcement learning model for trajectory planning of a robotic arm, selected based on environmental obstacle distribution label information, may include:

[0074] Step S403: If the environmental obstacle distribution label information is sparse obstacle distribution label, configure the trajectory planning reinforcement learning model of the robotic arm as a sparse trajectory planning reinforcement learning model.

[0075] Step S404: If the environmental obstacle distribution label information is a dense obstacle distribution label, configure the trajectory planning reinforcement learning model of the robotic arm as a dense trajectory planning reinforcement learning model.

[0076] In an optional embodiment of this application, please refer to Figure 4 The reinforcement learning-based robotic arm trajectory planning method may also include:

[0077] Step S406: Input the transferable trajectory information of the robotic arm with sparse obstacle distribution labels into the dense trajectory planning reinforcement learning model to generate obstacle avoidance trajectory information of the robotic arm.

[0078] In step S407, in response to the obstacle distribution label information of the robotic arm being sparsely labeled with obstacle distribution, the robotic arm identifies obstacles in the real-time trajectory corresponding to the real-time trajectory information during its movement under the control of the real-time trajectory information, and updates the real-time trajectory information based on the obstacle avoidance trajectory information.

[0079] In one exemplary embodiment of this application, such as Figure 4 As shown, a reinforcement learning-based robotic arm trajectory planning method is provided, which may include:

[0080] Step S401: Obtain the real-time pose parameters, target pose parameters, and state transition parameters of the robotic arm.

[0081] Step S402: Based on real-time pose parameters, state transition parameters and target pose parameters, transferable trajectory information is identified from the historical trajectory database.

[0082] Step S403: If the environmental obstacle distribution label information is sparse obstacle distribution label, configure the trajectory planning reinforcement learning model of the robotic arm as a sparse trajectory planning reinforcement learning model.

[0083] Step S404: If the environmental obstacle distribution label information is a dense obstacle distribution label, configure the trajectory planning reinforcement learning model of the robotic arm as a dense trajectory planning reinforcement learning model.

[0084] Step S405: Input the transferable trajectory information into the trajectory planning reinforcement learning model that matches the environmental obstacle distribution label information to generate the real-time trajectory information of the robotic arm.

[0085] Step S406: Input the transferable trajectory information of the robotic arm with sparse obstacle distribution labels into the dense trajectory planning reinforcement learning model to generate obstacle avoidance trajectory information of the robotic arm.

[0086] In step S407, in response to the obstacle distribution label information of the robotic arm being sparsely labeled with obstacle distribution, the robotic arm identifies obstacles in the real-time trajectory corresponding to the real-time trajectory information during its movement under the control of the real-time trajectory information, and updates the real-time trajectory information based on the obstacle avoidance trajectory information.

[0087] In the aforementioned reinforcement learning-based robotic arm trajectory planning method, by identifying transferable trajectory information from the historical trajectory database and inputting it into a trajectory planning reinforcement learning model that matches the distribution label information of environmental obstacles, a real-time trajectory adapted to the current task can be quickly generated. This achieves efficient, accurate, and adaptable robotic arm trajectory planning in different environments, thereby improving the efficiency, adaptability, and accuracy of robotic arm trajectory planning in complex environments, and enhancing the production efficiency and real-time performance of robotic arm task execution.

[0088] Furthermore, in the aforementioned reinforcement learning-based robotic arm trajectory planning method, by configuring the robotic arm's trajectory planning reinforcement learning model as a sparse trajectory planning reinforcement learning model or a dense trajectory planning reinforcement learning model according to the environmental obstacle distribution label information, adaptive trajectory planning of the robotic arm under different obstacle distribution environments can be achieved, improving the accuracy and adaptability of trajectory planning. By inputting the transferable trajectory information of the robotic arm with sparse obstacle distribution labels into the dense trajectory planning reinforcement learning model to generate obstacle avoidance trajectory information for the robotic arm, the ability of the robotic arm to quickly avoid unexpected obstacles in sparse environments can be improved, enhancing the stability and reliability of the robotic arm in dynamically changing environments. By updating real-time trajectory information based on obstacle avoidance trajectory information, real-time trajectory adjustment and optimization of the robotic arm during movement can be achieved, enabling the robotic arm to operate stably in dynamically changing environments, improving the intelligence level and task execution efficiency of the robotic arm.

[0089] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0090] Based on the same inventive concept, this application also provides a reinforcement learning-based robotic arm trajectory planning device for implementing the reinforcement learning-based robotic arm trajectory planning method described above. The solution provided by this device is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more reinforcement learning-based robotic arm trajectory planning device embodiments provided below can be found in the limitations of the reinforcement learning-based robotic arm trajectory planning method described above, and will not be repeated here.

[0091] In one exemplary embodiment, such as Figure 5 As shown, a reinforcement learning-based robotic arm trajectory planning device 500 is provided, which may include:

[0092] The motion parameter acquisition module 501 can be used to acquire the real-time pose parameters, target pose parameters, and state transition parameters of the robotic arm.

[0093] The transferable trajectory recognition module 502 can be used to identify transferable trajectory information from a historical trajectory database based on real-time pose parameters, state transition parameters and target pose parameters.

[0094] The environmental parameter acquisition module 503 can be used to acquire the environmental obstacle distribution label information of the robotic arm, and select the trajectory planning reinforcement learning model of the robotic arm based on the environmental obstacle distribution label information.

[0095] The real-time trajectory generation module 504 can be used to input transferable trajectory information into a trajectory planning reinforcement learning model that matches the distribution label information of environmental obstacles, thereby generating real-time trajectory information of the robotic arm.

[0096] In an optional embodiment of this application, the motion parameter acquisition module 501 can also be used for:

[0097] Based on the state transition parameters, the historical trajectory approximating the state is selected from the historical trajectory database, and the trajectory control vector of the historical trajectory approximating the state is obtained.

[0098] The target trajectory control vector of the robotic arm is constructed based on real-time pose parameters and target pose parameters.

[0099] Calculate the error distance between the target trajectory control vector of the robotic arm and the trajectory control vector of the historical trajectory that approximates the state, and generate trajectory difference parameters.

[0100] Based on the trajectory difference parameter, the transferable trajectory is identified from the state approximation historical trajectory, and the trajectory information of the transferable trajectory is set as the transferable trajectory information.

[0101] In an optional embodiment of this application, the motion parameter acquisition module 501 may further be used for:

[0102] Identify historical trajectories whose trajectory difference parameter is greater than the preset trajectory similarity threshold as pre-selected transferable trajectories.

[0103] Set the pre-selected transferable trajectory with the largest trajectory difference parameter as the transferable trajectory.

[0104] In an optional embodiment of this application, the motion parameter acquisition module 501 can also be used for:

[0105] The state transition parameter difference of historical trajectories in the historical trajectory database is calculated based on structural state parameters and performance degradation parameters.

[0106] Historical trajectories with state transition parameter differences less than a preset state transition parameter difference threshold are selected as approximate historical trajectories.

[0107] In an optional embodiment of this application, the environmental parameter acquisition module 503 may also be used for:

[0108] If the environmental obstacle distribution label information is sparse obstacle distribution label, configure the robot arm's trajectory planning reinforcement learning model as a sparse trajectory planning reinforcement learning model.

[0109] If the environmental obstacle distribution label information is a dense obstacle distribution label, configure the robot arm's trajectory planning reinforcement learning model as a dense trajectory planning reinforcement learning model.

[0110] In an optional embodiment of this application, the reinforcement learning-based robotic arm trajectory planning device 500 can also be used for:

[0111] The transferable trajectory information of the robotic arm with sparse obstacle distribution labels is input into the dense trajectory planning reinforcement learning model to generate the obstacle avoidance trajectory information of the robotic arm.

[0112] In response to the environmental obstacle distribution label information, the robotic arm with sparse obstacle distribution labels identifies obstacles in the real-time trajectory corresponding to the real-time trajectory information during its movement under the control of real-time trajectory information, and updates the real-time trajectory information based on the obstacle avoidance trajectory information.

[0113] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a reinforcement learning-based robotic arm trajectory planning method as described above.

[0114] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0115] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0116] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for trajectory planning of a robot arm based on reinforcement learning, characterized in that, The method comprises: obtaining real-time pose parameters, target pose parameters and state transition parameters of a mechanical arm; identifying transferable trajectory information from a historical trajectory database based on the real-time pose parameters, the state transition parameters and the target pose parameters; obtaining environment obstacle distribution label information of the mechanical arm, and selecting a trajectory planning reinforcement learning model of the mechanical arm based on the environment obstacle distribution label information; inputting the transferable trajectory information into the trajectory planning reinforcement learning model matched with the environment obstacle distribution label information to generate real-time trajectory information of the mechanical arm.

2. The method of claim 1, wherein, The method comprises: filtering state-approximate historical trajectories from the historical trajectory database based on the state transition parameters, and obtaining trajectory control vectors of the state-approximate historical trajectories; constructing target trajectory control vectors of the mechanical arm based on the real-time pose parameters and the target pose parameters; calculating error distances between the target trajectory control vectors of the mechanical arm and the trajectory control vectors of the state-approximate historical trajectories to generate trajectory difference parameters; identifying transferable trajectories from the state-approximate historical trajectories according to the trajectory difference parameters, and setting trajectory information of the transferable trajectories as the transferable trajectory information.

3. The method of claim 2, wherein, The method comprises: identifying preselected transferable trajectories from the state-approximate historical trajectories according to the trajectory difference parameters; setting the preselected transferable trajectory with the largest trajectory difference parameter as the transferable trajectory.

4. The method of claim 3, wherein, The expression of the trajectory difference parameter is: In the formula, P TS is the trajectory similarity parameter of the target trajectory, is the target trajectory control vector of the target trajectory, is the trajectory control vector of the target trajectory, and are respectively a real-time pose matrix corresponding to the real-time pose parameter of the robot arm and a target pose matrix corresponding to the target pose parameter of the robot arm, T TTC is a trajectory time parameter of the robot arm, and are respectively the real-time pose matrix of the state-approximate historical trajectory and the target pose matrix of the state-approximate historical trajectory, T TC is the trajectory time parameter of the state-approximate historical trajectory.

5. The method of claim 2, wherein, The state transition parameters comprise structure state parameters and performance degradation parameters, and the method comprises: calculating state transition parameter differences of historical trajectories in the historical trajectory database based on the structure state parameters and the performance degradation parameters; filtering the historical trajectories with state transition parameter differences less than a preset state transition parameter difference threshold as the state-approximate historical trajectories.

6. The method according to any one of claims 1 to 5, characterized in that, The environment obstacle distribution label information comprises obstacle distribution sparse labels and obstacle distribution dense labels, the trajectory planning reinforcement learning model comprises a sparse trajectory planning reinforcement learning model and a dense trajectory planning reinforcement learning model, target functions of the sparse trajectory planning reinforcement learning model and the dense trajectory planning reinforcement learning model comprise trajectory obstacle avoidance target functions, a trajectory obstacle avoidance target function weight of the trajectory obstacle avoidance target function of the sparse trajectory planning reinforcement learning model is lower than a trajectory obstacle avoidance target function weight of the trajectory obstacle avoidance target function of the dense trajectory planning reinforcement learning model, and the method comprises: If the environmental obstacle distribution label information is the obstacle distribution sparse label, the trajectory planning reinforcement learning model of the robotic arm is configured as the sparse trajectory planning reinforcement learning model; If the environmental obstacle distribution label information is the dense obstacle distribution label, the trajectory planning reinforcement learning model of the robotic arm is configured as the dense trajectory planning reinforcement learning model.

7. The method of claim 6, wherein, The method further includes: The transferable trajectory information of the robotic arm with sparse obstacle distribution label information is input into the dense trajectory planning reinforcement learning model to generate the obstacle avoidance trajectory information of the robotic arm. In response to the environmental obstacle distribution label information being the obstacle distribution sparse label, the robotic arm, under the control of the real-time trajectory information, identifies obstacles in the real-time trajectory corresponding to the real-time trajectory information during its movement, and updates the real-time trajectory information based on the obstacle avoidance trajectory information.

8. A reinforcement learning-based trajectory planning device for a robot arm, characterized by, The device includes: The motion parameter acquisition module is used to acquire the real-time pose parameters, target pose parameters, and state transition parameters of the robotic arm; A transferable trajectory recognition module is used to identify transferable trajectory information from a historical trajectory database based on the real-time pose parameters, the state transition parameters, and the target pose parameters. An environmental parameter acquisition module is used to acquire environmental obstacle distribution label information of the robotic arm, and select a trajectory planning reinforcement learning model for the robotic arm based on the environmental obstacle distribution label information. The real-time trajectory generation module is used to input the transferable trajectory information into the trajectory planning reinforcement learning model that matches the environmental obstacle distribution label information, thereby generating the real-time trajectory information of the robotic arm. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images