Kalman filtering trajectory optimization method based on reinforcement learning

Abstract

This invention relates to a Kalman filter trajectory optimization method based on reinforcement learning, comprising the following steps: establishing a kinematic model in Cartesian space to construct a robotic arm motion space; optimizing the trajectory of the established motion model using deep reinforcement learning and federated Kalman filtering; building a six-axis robot working experimental platform for application testing; setting the motion trajectory of the robotic arm, exporting encoder data during the robotic arm's motion process, the encoder data reflecting the actual motion state of each joint of the robotic arm during the motion process; comparing the optimized trajectory with the actual motion state, and performing training and further optimization; this invention optimizes the trajectory of an industrial six-axis robot by combining deep reinforcement learning and federated Kalman filtering, and through techniques such as discretization, deep reinforcement learning, and federated Kalman filtering algorithms, makes the robot's motion more in line with actual needs, and has better performance and accuracy.

Description

Technical Field

[0001] This invention relates to the field of robot trajectory optimization, and more particularly to a Kalman filter trajectory optimization method based on reinforcement learning. Background Technology

[0002] Since humanity entered the era of "Industry 4.0," the digitalization of the entire product lifecycle and manufacturing process, along with module integration based on information and communication technologies, will create a highly flexible, personalized, and digitalized production model for products and services. In the coming years, the focus will be on 10 key application areas, achieving breakthroughs in over 100 innovative robot application technologies and solutions, promoting over 200 typical robot application scenarios with high technical levels, innovative application models, and significant application results, cultivating a number of "robot+" application benchmark enterprises, and building a number of application experience centers and testing and verification centers. Furthermore, efforts will be made to encourage various industries and regions to conduct innovative "robot+" application practices based on their industry development stages and regional characteristics.

[0003] Upgrading industries to be more intelligent will help enterprises improve manufacturing efficiency and quality, thereby enhancing their overall competitiveness. With the rapid development and widespread adoption of sensing technology and next-generation information technology, multi-sensor-based motion monitoring and control technology has become mainstream in the field of intelligent manufacturing. By collecting information and enhancing interaction with uncertain environments, intelligent manufacturing equipment gains the ability to perceive the environment and make intelligent decisions, enabling intelligent multi-scenario applications. However, traditional algorithms cannot meet the high dynamic accuracy requirements during motion. Currently, six-axis robots can replace manual labor in most repetitive, complex, and dangerous tasks in the manufacturing sector, playing a significant role in promoting new industrialization and accelerating the construction of a manufacturing powerhouse. Therefore, focusing on six-axis industrial robots, there is an urgent need to develop Kalman filter trajectory optimization technology based on reinforcement learning to achieve dynamic and accurate trajectory tracking of industrial robots, thus contributing to high-precision intelligent production in the new industrialization and intelligent manufacturing industry. Summary of the Invention

[0004] The purpose of this invention is to provide a Kalman filter trajectory optimization method based on reinforcement learning. This method optimizes the trajectory of an industrial six-axis robot by combining deep reinforcement learning and federated Kalman filtering. Through techniques such as discretization, deep reinforcement learning, and federated Kalman filtering algorithms, the robot's motion trajectory is optimized, making the robot's motion more in line with actual needs and achieving better performance and accuracy.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: a Kalman filter trajectory optimization method based on reinforcement learning, comprising the following steps: Step 1: Establish its kinematic model in Cartesian space to construct the motion space of the robotic arm; Step 2: Optimize the trajectory of the motion model established in Step 1 using deep reinforcement learning and federated Kalman filtering; Step 3: A six-axis robot working experimental platform was built for application testing; Step 4: The motion trajectory of the robotic arm was set, and the encoder data during the movement of the robotic arm was exported to Excel through the Modbus-TCP network port and RS485 serial port. This encoder data reflects the actual motion state of each joint of the robotic arm during the movement process. Step 5: Compare the optimized trajectory with the actual motion state. If it meets the set error requirements, the final optimized trajectory is obtained. If it does not meet the requirements, repeat steps 2-4 for training and further optimization.

[0006] Preferably, step one specifically involves: performing kinematic and dynamic analysis on the robotic arm, that is, firstly planning the trajectory according to the task requirements through the robotic arm controller, and then sending the planned data at each moment to the motors of each joint of the robotic arm, simultaneously energizing and driving their movement. During trajectory planning, the running trajectory is modeled by combining kinematic error analysis.

[0007] Preferably, when performing trajectory modeling, the coordinate system for robot motion is first defined. The first step is to use a Cartesian coordinate system to describe the position and orientation of the robot's end effector. The origin of the coordinate system will be located at the bottom of the robot, and the x, y, and z axes will be aligned with the robot's frame. The orientation of the end effector is represented by Euler angles, which describe the rotation of the end effector around each of the three axes in the coordinate system. The second step is to determine the range of motion of each joint of the robot and the maximum angle that each joint can rotate. This information is used to define the joint angles of the six joints in the robot arm.

[0008] Preferably, when performing trajectory modeling, an inverse kinematics model is adopted, that is, given the relative pose of the end effector of the robotic arm in space with respect to the polar coordinate system, the angle that each joint should rotate under this pose is solved, specifically using closed-loop solution method and numerical solution method.

[0009] Preferably, the motion errors of a six-axis robot include mechanical structure errors, sensor errors, control system errors, environmental errors, and software algorithm errors. Preferably, in step two, deep reinforcement learning utilizes deep learning algorithms to model high-dimensional state and action spaces, improving the robot's learning efficiency and performance. In the application of industrial six-axis robots, deep reinforcement learning achieves optimization through the following steps: 1) Define the state space: Based on the robot's motion model and sensor information, define the robot's state space, including the robot's position, velocity, acceleration, and joint angle information; 2) Define the action space: Define the robot's action space, i.e., the actions the robot can take, including the robot's joint movements, the movement of the end effector, and the rotation of the tool; 3) Define the reward function: Define the reward obtained by the robot after performing a certain action, which serves as the target for the robot's learning; 4) Train a deep neural network: Use deep learning algorithms to train a neural network model, mapping the robot's state and actions to the reward function, thereby achieving the learning and optimization of the robot.

[0010] Preferably, federated Kalman filtering is used to estimate and predict the robot's state, reducing the robot's positioning error. The specific steps are as follows: 1) Collect distributed sensor data: In industrial six-axis robots, sensors are used to measure the robot's position, velocity, and acceleration information. By collecting data from multiple sensors, more comprehensive and accurate information about the robot's state can be obtained; 2) Perform distributed Kalman filtering: Distributed Kalman filtering is performed on the data from multiple sensors to obtain the state estimation result of each sensor; 3) Merge the state estimation results: The state estimation results of multiple sensors are merged to obtain the final state estimation result of the robot.

[0011] Preferably, the six-axis robot working experimental platform includes an industrial six-axis robot, a robot servo driver, a robot controller, a robot mounting base, an operating table, a computer, robot cables, and a binocular camera. The binocular camera is used to capture the robot's motion trajectory in space and to perceive the robot's motion. The camera system communicates with the computer through various communication interfaces.

[0012] Preferably, in the experimental data acquisition process, a target object is placed in the robot's workspace to simulate the actual working environment. Then, the robot is remotely controlled via a control panel to execute a predetermined trajectory. Simultaneously, a binocular camera collects the motion state information of the robot's end effector and saves this information. Using MATLAB software, the collected robot state information is processed and analyzed to extract key information such as the robot's current state and target position, which is used to calculate the robot's current strategy. The robot's trajectory is optimized using an implemented deep reinforcement learning algorithm to obtain the strategy for executing the robot's trajectory. The optimized trajectory is then sent to the robot for execution via a controller. The robot executes the optimized trajectory according to the instructions sent by the controller, and the binocular camera collects the robot's state information in real time, which is then sent to a computer for processing and analysis.

[0013] Preferably, in the experiment, a deep reinforcement learning algorithm was used to optimize the robot's trajectory. A policy gradient-based deep reinforcement learning algorithm was employed, and the algorithm code was implemented using the PyTorch framework. To implement the federated Kalman filter algorithm, MATLAB software was also used. Specifically, the robot's target trajectory was first decomposed into a series of discrete time points, i.e., discretized. Then, the robot's state information, such as current position, velocity, and acceleration, was input into the deep reinforcement learning network. The neural network was used to control the robot's motion. Through the deep reinforcement learning algorithm, the robot's motion trajectory was optimized to meet actual requirements and achieve better performance and accuracy. Furthermore, the federated Kalman filter algorithm was used to estimate and optimize the robot's state. This algorithm uses the robot's state information and motion model to predict the robot's motion trajectory and improves the robot's motion accuracy and stability through continuous iterative optimization.

[0014] The technical effects of this invention are as follows: 1. Trajectory optimization of industrial six-axis robots is achieved by combining deep reinforcement learning and federated Kalman filtering. Through techniques such as discretization, deep reinforcement learning, and federated Kalman filtering algorithms, the robot's motion trajectory is optimized, making the robot's motion more in line with actual needs and achieving better performance and accuracy.

[0015] 2. Based on deep reinforcement learning algorithms, a robot motion control model is established to achieve adaptive control. By optimizing the control strategy, the robot's errors and oscillations are reduced, and the robot's motion accuracy and stability are improved. In terms of federated Kalman filtering, the robot's sensor data will be analyzed and processed. By establishing a state-space model, the robot's position and velocity will be estimated, thereby correcting the robot's trajectory.

[0016] 3. In the test experiment, the trajectory of the industrial six-axis robot was optimized by combining deep reinforcement learning and federated Kalman filtering. Through discretization, deep reinforcement learning and federated Kalman filtering algorithms, the robot's motion trajectory was optimized to make the robot's motion more in line with actual needs, with better performance and accuracy. This also allows for better cooperation with reinforcement learning to further optimize the motion trajectory model. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a Kalman filter trajectory optimization method based on reinforcement learning. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings. The description in this part is only exemplary and explanatory, and should not be used to limit the scope of protection of the present invention in any way.

[0019] like Figure 1 As shown, this application presents a Kalman filter trajectory optimization method based on reinforcement learning, comprising the following steps: Step 1: Establish its kinematic model in Cartesian space to construct the motion space of the robotic arm; Step 2: Optimize the trajectory of the motion model established in Step 1 using deep reinforcement learning and federated Kalman filtering; Step 3: A six-axis robot working experimental platform was built for application testing; Step 4: The motion trajectory of the robotic arm was set, and the encoder data during the movement of the robotic arm was exported to Excel through the Modbus-TCP network port and RS485 serial port. This encoder data reflects the actual motion state of each joint of the robotic arm during the movement process. Step 5: Compare the optimized trajectory with the actual motion state. If it meets the set error requirements, the final optimized trajectory is obtained. If it does not meet the requirements, repeat steps 2-4 for training and further optimization.

[0020] The following details steps one through five. 1. Analysis of Robot Kinematics and Its Characteristics 1.1 Kinematics of a Serial Robotic Arm: For a robotic arm to complete a task, the controller first plans the trajectory according to the task requirements, then sends the data for each moment of the plan to the motors of each joint of the robotic arm, simultaneously energizing and driving their movement. Trajectory planning is indispensable in this process, and kinematic and dynamic analysis is the foundation of trajectory planning; therefore, kinematic and dynamic analysis of the robotic arm must be performed first.

[0021] 1.2 Establishment of the inverse kinematics model: The inverse kinematics problem of the robotic arm is to solve the angles that each joint should rotate under the given relative pose of the end effector of the robotic arm in space with respect to the polar coordinate system.

[0022] In inverse kinematics problems, there may be situations where multiple motion methods can reach the specified position and posture, which are multi-solution problems in inverse kinematics. However, there may also be situations where the specified posture is outside the working range of the robotic arm, which are unsolvable problems. In such cases, it is necessary to rely on the solution of inverse kinematics to infer whether the specified posture is reasonable. Therefore, the solution of inverse kinematics problems has an important impact on the control of robotic arms. The control system of a robotic arm can only use one set of inverse solutions to drive the rotation of each joint motor. Therefore, when there are multiple solutions, it is necessary to use certain selection criteria, such as the shortest path, to select the optimal solution suitable for the current working condition.

[0023] Inverse kinematics problems of robotic arms typically involve two solution methods: (1) Closed-loop solution method, also known as analytical solution method, expresses the solution using analytical functions. It is faster to solve and can also ensure good accuracy. When designing a robotic arm, having a closed-loop solution is generally taken as the design goal. The robotic arm only has a closed-loop solution under certain specific conditions, such as the existence of multiple joint axes with orthogonal relationships or multiple links with rotation angles of 0 or 90°. (2) Numerical solution: Numerical solution requires multiple iterations, but the number of iterations is difficult to control, the accuracy is uncertain, and there may be singular solutions. The solution speed is generally slower than the corresponding closed solution.

[0024] 1.3 Motion error analysis of an industrial six-axis robot, mainly including the following errors Mechanical structure errors: Due to the influence of manufacturing processes and human factors on the machining and assembly accuracy of mechanical structures, there are certain errors inherent in the mechanical structure itself. In addition, the limitations of the rigidity and torque transmission performance of mechanical structures can also lead to mechanical structure errors. These errors will affect the positioning accuracy and stability of the robot and need to be reduced through mechanical structure optimization.

[0025] Sensor errors: Commonly used sensors in robots include encoders, force sensors, and vision sensors, all of which inherently possess errors. Encoder accuracy is affected by factors such as resolution, signal-to-noise ratio, and interference; force sensor accuracy is affected by factors such as sensitivity and rigidity; and vision sensor accuracy is affected by factors such as resolution, ambient light, and illumination. These errors affect the machine's perception and measurement accuracy and need to be reduced through sensor optimization.

[0026] Control system errors: The robot's control system consists of controllers, drivers, encoders, etc. The accuracy of the controller is affected by factors such as sampling frequency and calculation accuracy. The accuracy of the driver is affected by factors such as output waveform and current sampling. The accuracy of the encoder is affected by factors such as signal processing and noise. These errors will affect the robot's control accuracy and stability and need to be reduced through control system optimization. Environmental errors: The working environment of a robot can also affect its positioning accuracy and stability. Factors such as temperature, humidity, and light can affect the accuracy of the robot's sensors and the performance of its mechanical structure. Dust and foreign objects can affect the robot's motion performance. These errors need to be reduced through environmental optimization.

[0027] Software algorithm errors: Software algorithms such as trajectory planning and motion control of robots can also affect their positioning accuracy and stability. The accuracy and calculation speed of trajectory planning algorithms will affect the accuracy and smoothness of the robot's trajectory, while the response speed and anti-interference ability of motion control algorithms will affect the control accuracy and stability of the robot. These errors need to be reduced through software algorithm optimization.

[0028] In general, the sources of error in industrial six-axis robots are diverse, requiring optimization from multiple aspects such as mechanical structure, sensors, control systems, environment, and software algorithms to achieve higher positioning accuracy and stability.

[0029] The first step in constructing the motion space is to define the coordinate system that will be used to describe the robot's motion.

[0030] The first step is to use a Cartesian coordinate system to describe the position and orientation of the robot's end effector. The origin of the coordinate system will be located at the bottom of the robot, and the x, y, and z axes will be aligned with the robot's base. The orientation of the end effector will be represented using Euler angles, which describe the rotation of the end effector about each of the three axes in the coordinate system.

[0031] The second step is to determine the range of motion of each joint of the robot and the maximum angle that each joint can rotate; use this information to define the joint angles of the six joints in the robot arm.

[0032] 2. Optimization Methods To optimize the robot's trajectory, reinforcement learning is used. This involves training the robot to perform tasks through trial and error. A joint Kalman filter is used to estimate the robot's state during the learning process. This allows the robot to track its motion and adjust its trajectory as needed to achieve optimal performance.

[0033] 2.1 Motion Trajectory Optimization Algorithm: This invention optimizes the trajectory of an industrial six-axis robot from two perspectives: deep reinforcement learning and federated Kalman filtering. Regarding deep reinforcement learning, this invention establishes a robot motion control model based on reinforcement learning algorithms to achieve adaptive control. By optimizing the control strategy, it reduces robot errors and oscillations, improving the robot's motion accuracy and stability. Regarding federated Kalman filtering, this invention analyzes and processes the robot's sensor data, establishes a state-space model, estimates the robot's position and velocity, and thus corrects the robot's trajectory.

[0034] This invention aims to provide a new method for adaptive control and motion trajectory optimization of industrial robots, thereby improving the accuracy and stability of robots and promoting the development of industrial production.

[0035] 2.2 Deep Reinforcement Learning (DRL) is an extension of reinforcement learning that utilizes deep learning algorithms to model high-dimensional state and action spaces, thereby improving the learning efficiency and performance of robots. In the application of industrial six-axis robots, DRL can be optimized through the following steps: Define the state space: Based on the robot's motion model and sensor information, define the robot's state space; these states include information such as the robot's position, velocity, acceleration, and joint angles.

[0036] Define the motion space: Define the robot's motion space, that is, the actions the robot can take; these actions can include joint movements, end effector movement, tool rotation, and other operations.

[0037] Define the reward function: Define the reward that the robot receives after performing a certain action, and use this as the learning target of the robot. The design of the reward function is based on the requirements of the task and the capabilities of the robot.

[0038] Training a deep neural network: Using deep learning algorithms, a neural network model is trained to map the robot's state and actions to a reward function, thereby enabling the robot to learn and optimize. Applying deep reinforcement learning algorithms: In practical applications, the trained deep neural network is applied to the robot control system to optimize the robot.

[0039] DRL has the ability to learn and optimize autonomously, which can effectively improve the performance and efficiency of robots. In the application of industrial six-axis robots, DRL can be used to optimize the robot's path planning, motion control, obstacle avoidance and other aspects to improve the robot's precision and accuracy.

[0040] 2.3 Federated Kalman Filter: Estimates and predicts the robot's state, reducing the robot's positioning error.

[0041] Specifically, federated Kalman filtering can be performed through the following steps: 1) Collecting distributed sensor data: In industrial six-axis robots, sensors can be used to measure information such as the robot's position, speed, and acceleration; by collecting data from multiple sensors, more comprehensive and accurate information about the robot's status can be obtained.

[0042] 2) Perform distributed Kalman filtering: Perform distributed Kalman filtering on the data from multiple sensors to obtain the state estimation results for each sensor.

[0043] 3) Merge state estimation results: Merge the state estimation results of multiple sensors to obtain the final state estimation result of the robot.

[0044] 2.4 The correspondence of key elements of reinforcement learning in industrial six-axis robot application scenarios; A. Determining the basic elements: In reinforcement learning, the agent needs to achieve the goal in a changing environment. Different application scenarios require different agents, environments, and goals to be determined based on the expected objectives.

[0045] 1) Determine the agent: In this invention, the robot end effector is used as the agent to study the Kalman filter trajectory optimization model based on reinforcement learning.

[0046] 2) Environment construction: In this invention, the environment is the three-dimensional space in which the intelligent agent moves.

[0047] 3) Determine the objective: In this invention, the objective is to optimize the robot's end effector trajectory in both time and space dimensions, i.e., the objective elements are to optimize the agent's motion time and minimize the trajectory error.

[0048] B. Identify the main elements 1) Determine the state: The state is the agent's position in the environment, denoted as . 2) Determine the action: An action is the action taken by an agent to transition from one state to another, denoted as . .

[0049] 3) Determine the reward: The reward is a feedback from the environment to the agent, denoted as... This means that in the state The agent takes action The average reward obtained at that time.

[0050] C. Identify core elements 1) Determine the strategy: Adopt Greedy algorithms are used as the strategy in the reinforcement learning method of this invention.

[0051] 2) Determine the value: In time-optimal trajectory optimization research, the value judgment is based on the time it takes for the agent to complete a set of actions. The faster it completes, the higher the value, and the slower it completes, the lower the value.

[0052] 2.5 Determine the Federal Kalman Filter Data: After the Kalman filter data is determined, it is filtered to achieve spatial optimization of the robot's end effector motion trajectory.

[0053] A. Kalman filter information allocation: The global state, variance, and system noise are allocated to the main filter and local filters through allocation factors.

[0054] B. Kalman filter time update: The main filter and local filters are updated independently.

[0055] C. Kalman filter measurement update: The measurement information of the corresponding local filter is updated on both sides using the sub-sensor.

[0056] D. Optimal information fusion of Kalman filter: The estimation information of the main filter and the local filter is fused into new global state estimation information.

[0057] E. Research on establishing a Kalman filter trajectory optimization model based on reinforcement learning: to achieve optimization of the end-effector trajectory in both time and space dimensions.

[0058] 3. Experimental Testing 3.1 Setting up the experimental platform: Setting up the experimental platform is a crucial step in this invention. A complete experimental platform is constructed to facilitate experiments on deep reinforcement learning and federated Kalman filtering. The equipment used is as follows: Industrial Six-Axis Robot: A 2m reach, 10kg payload industrial six-axis robot is used. This robot achieves high precision and flexibility, making it ideal for the experimental purposes of this invention. Robot Servo Drivers: Six robot servo drivers are selected to drive the robot's six degrees of freedom. These drivers communicate with the computer through various interfaces and provide powerful control functions. Robot Controller: A robot controller is used to achieve motion control and data acquisition for the robot. This controller communicates with the computer through various communication interfaces. Robot Mounting Base: Provides a fixed foundation structure for the robot to ensure its motion stability and safety.

[0059] Control panel: A control panel is required to operate the robot and monitor the experimental process. The control panel should have sufficient workspace and be designed in accordance with ergonomic principles.

[0060] Computer: A high-performance computer is used as the main control device of the experimental platform. This computer can communicate with the robot controller and camera through various communication interfaces. It also has powerful computing capabilities and can complete experimental tasks of deep reinforcement learning and federated Kalman filtering.

[0061] Binocular camera: A binocular camera system is selected to capture the robot's movement trajectory in space and perceive the robot's movement. The camera system can communicate with the computer through various communication interfaces; Complete robot cable set: A complete set of robot cables is required to ensure the robot's power supply and signal transmission. When setting up the experimental platform, the robot was first mounted on a fixed base, and the actuators were connected to the robot's various degrees of freedom. Next, the robot controller and computer were connected, and communication tests were conducted to ensure normal communication between them. Then, the binocular camera system was mounted on the control panel and connected to the computer, while communication tests were performed. 3.2 Collecting Experimental Data First, target objects need to be placed in the robot's workspace to simulate the actual working environment. Then, the robot is remotely controlled via a control panel to execute a predetermined trajectory. Simultaneously, a binocular camera collects the motion state information of the robot's end effector and saves this information. Using MATLAB software, the collected robot state information is processed and analyzed to extract key information such as the robot's current state and target position, which is used to calculate the robot's current strategy. Using a implemented deep reinforcement learning algorithm, the robot's trajectory is optimized to obtain the strategy for executing the trajectory. The optimized trajectory is then sent to the robot for execution via the controller. The robot executes the optimized trajectory according to the instructions sent by the controller, and the binocular camera collects the robot's state information in real time, which is then sent to a computer for processing and analysis.

[0062] 3.3 Trajectory Optimization In the experiment, deep reinforcement learning algorithms were used to optimize the robot's trajectory. Specifically, a policy gradient-based deep reinforcement learning algorithm was adopted, and the algorithm code was implemented using the PyTorch framework. In addition, MATLAB software was used to implement the federated Kalman filter algorithm.

[0063] In this experiment, the robot's target trajectory is first decomposed into a series of discrete time points, i.e., discretized. Then, the robot's state information, such as current position, velocity, and acceleration, is input into a deep reinforcement learning network. The neural network is used to control the robot's motion. Through reinforcement learning algorithms, such as deep Q-learning or policy gradient algorithms, the robot's motion trajectory is optimized to make the robot's motion more in line with actual needs and achieve better performance and accuracy.

[0064] In addition, to improve the robot's motion accuracy and stability, this experiment also adopted the federated Kalman filter algorithm to estimate and optimize the robot's state. This algorithm can use the robot's state information and motion model to predict the robot's motion trajectory, and improve the robot's motion accuracy and stability through continuous iterative optimization.

[0065] In summary, this experiment optimized the trajectory of an industrial six-axis robot by combining deep reinforcement learning and federated Kalman filtering. Through techniques such as discretization, deep reinforcement learning, and federated Kalman filtering, the robot's motion trajectory was optimized, making the robot's motion more in line with actual needs and achieving better performance and accuracy.

[0066] It should be noted that, in this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0067] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only for the purpose of helping to understand the method and core ideas of the present invention. The above descriptions are only preferred embodiments of the present invention. It should be noted that due to the limitations of textual expression, while there are objectively infinite specific structures, those skilled in the art can make several improvements, modifications, or changes without departing from the principles of the present invention, and can also combine the above technical features in an appropriate manner. These improvements, modifications, changes, or combinations, or the direct application of the inventive concept and technical solution to other situations without modification, should all be considered within the scope of protection of the present invention.

Claims

1. A Kalman filter trajectory optimization method based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Establish its kinematic model in Cartesian space to construct the motion space of the robotic arm; Step 2: Optimize the trajectory of the motion model established in Step 1 using deep reinforcement learning and federated Kalman filtering; Step 3: A six-axis robot working experimental platform was built for application testing; Step 4: The motion trajectory of the robotic arm was set, and the encoder data during the movement of the robotic arm was exported to Excel through the Modbus-TCP network port and RS485 serial port. This encoder data reflects the actual motion state of each joint of the robotic arm during the movement process. Step 5: Compare the optimized trajectory with the actual motion state. If it meets the set error requirements, the final optimized trajectory is obtained. If it does not meet the requirements, repeat steps 2-4 for training and further optimization.

2. The Kalman filter trajectory optimization method based on reinforcement learning according to claim 1, characterized in that, Step one specifically involves performing kinematic and dynamic analysis on the robotic arm. This means that the robotic arm controller first plans the trajectory according to the task requirements, then sends the data of each moment of the plan to the motors of each joint of the robotic arm, and simultaneously powers on and drives them to move. During trajectory planning, kinematic error analysis is combined to model the running trajectory.

3. The Kalman filter trajectory optimization method based on reinforcement learning according to claim 2, characterized in that, When performing trajectory modeling, the first step is to define the coordinate system for robot motion. The first step is to use a Cartesian coordinate system to describe the position and orientation of the robot's end effector. The origin of the coordinate system will be located at the bottom of the robot, and the x, y, and z axes will be aligned with the robot's frame. The orientation of the end effector is represented by Euler angles, which describe the rotation of the end effector around each of the three axes in the coordinate system. The second step is to determine the range of motion of each joint of the robot and the maximum angle that each joint can rotate. This information is used to define the joint angles of the six joints in the robot arm.

4. The Kalman filter trajectory optimization method based on reinforcement learning according to claim 3, characterized in that, When performing trajectory modeling, an inverse kinematics model is adopted, that is, given the relative pose of the end effector of the robotic arm in space with respect to the polar coordinate system, the angles that each joint should rotate under this pose are solved. Specifically, closed-loop solution method and numerical solution method are used.

5. A Kalman filter trajectory optimization method based on reinforcement learning according to any one of claims 2-4, characterized in that, The motion errors of a six-axis robot include mechanical structure errors, sensor errors, control system errors, environmental errors, and software algorithm errors.

6. The Kalman filter trajectory optimization method based on reinforcement learning according to claim 5, characterized in that, Step two, deep reinforcement learning, utilizes deep learning algorithms to model high-dimensional state and action spaces, improving the robot's learning efficiency and performance. In industrial six-axis robot applications, deep reinforcement learning achieves optimization through the following steps: 1) Define the state space: Based on the robot's motion model and sensor information, define the robot's state space, including the robot's position, velocity, acceleration, and joint angle information; 2) Define the action space: Define the robot's action space, i.e., the actions the robot can take, including joint movements, end effector movement, and tool rotation; 3) Define the reward function: Define the reward obtained by the robot after performing a certain action, using this as the robot's learning target; 4) Train the deep neural network: Using deep learning algorithms, train a neural network model to map the robot's state and actions to the reward function, thereby achieving robot learning and optimization.

7. The Kalman filter trajectory optimization method based on reinforcement learning according to claim 6, characterized in that, Federated Kalman filtering estimates and predicts the robot's state, reducing the robot's positioning error. The specific steps are as follows: 1) Collect distributed sensor data: In industrial six-axis robots, sensors are used to measure the robot's position, speed, and acceleration information. By collecting data from multiple sensors, more comprehensive and accurate information about the robot's state can be obtained. 2) Perform distributed Kalman filtering: Perform distributed Kalman filtering on the data from multiple sensors to obtain the state estimation results for each sensor; 3) Merge state estimation results: Merge the state estimation results from multiple sensors to obtain the final state estimation result of the robot.

8. The Kalman filter trajectory optimization method based on reinforcement learning according to claim 6, characterized in that, The six-axis robot working experimental platform includes an industrial six-axis robot, a robot servo driver, a robot controller, a robot mounting base, an operating table, a computer, robot cables, and a binocular camera. The binocular camera is used to capture the robot's motion trajectory in space and to perceive the robot's motion. The camera system communicates with the computer through various communication interfaces.

9. The Kalman filter trajectory optimization method based on reinforcement learning according to claim 8, characterized in that, Experimental data acquisition involved placing a target object within the robot's workspace to simulate the actual working environment. The robot was then remotely controlled via a control panel to execute a predetermined trajectory. Simultaneously, a binocular camera captured the motion state information of the robot's end effector, which was then saved. MATLAB software was used to process and analyze the acquired robot state information, extracting key information such as the robot's current state and target position to calculate the robot's current strategy. A implemented deep reinforcement learning algorithm was then used to optimize the robot's trajectory, resulting in a strategy for executing the trajectory. The optimized trajectory was then sent to the robot for execution via a controller. The robot executed the optimized trajectory according to the controller's instructions, while the binocular camera continuously captured the robot's state information in real time, sending it to a computer for processing and analysis.

10. The Kalman filter trajectory optimization method based on reinforcement learning according to claim 9, characterized in that, In the experiment, deep reinforcement learning algorithms were used to optimize the robot's trajectory. A policy gradient-based deep reinforcement learning algorithm was employed, and the algorithm code was implemented using the PyTorch framework. To implement the federated Kalman filter algorithm, MATLAB software was also used. Specifically, the robot's target trajectory was first decomposed into a series of discrete time points, i.e., discretized. Then, the robot's state information, such as current position, velocity, and acceleration, was input into the deep reinforcement learning network. The neural network was used to control the robot's motion. Through the deep reinforcement learning algorithm, the robot's trajectory was optimized to meet actual requirements and achieve better performance and accuracy. The federated Kalman filter algorithm was also used to estimate and optimize the robot's state. This algorithm uses the robot's state information and motion model to predict the robot's trajectory and improves the robot's motion accuracy and stability through continuous iterative optimization.

Images