A training-free natural language-driven robotic arm execution method based on a large language model

Abstract

This invention discloses a training-free natural language-driven robotic arm execution method based on a large language model, belonging to the field of intelligent robot technology. Addressing the problems of traditional robotic arm control requiring complex training, relying on manual pre-programming, and lacking flexibility in task planning, it provides an intelligent mapping method from natural language commands to robotic arm action execution. This includes a predefined skill atom library covering skills such as checking robotic arm status, interactive question answering, text-guided visual detection and localization based on YOLO-World+FastSAM, object grasping and placement, and object stacking; it employs the Qwen-Audio-ASR large model to achieve speech-to-text conversion; inputting the command text and prompts into the Qwen-Plus large language model to complete task reasoning, planning, and orchestration; and combining this with the robotic arm motion planning module to execute corresponding actions, ultimately achieving closed-loop control of command input, automatic planning, and task execution. This invention achieves a high degree of synergy between natural language understanding and robot task planning, enabling efficient and safe control of the robotic arm without complex training.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot technology, and in particular to a training-free natural language driven robotic arm execution method based on a large language model. Background Technology

[0002] With the current transformation and upgrading of the manufacturing industry and the rapid development of the service robot industry, the application of robotic arms has expanded from industrial assembly lines to complex fields such as medical assistance and daily life services. Traditional robotic arms rely on offline programming or teach-in control, which, while suitable for structured repetitive tasks, has significant limitations in scenarios with frequent task changes, dynamic environmental changes, and interactions with non-professional users. Traditional technologies mainly face two core problems: First, the flexibility of task planning is insufficient, requiring professionals with a foundation in robot kinematics and programming. Slight changes in the task or environment necessitate rewriting the control logic, resulting in low response efficiency. Second, the human-computer interaction threshold is high; existing command input methods are complex, and ordinary users cannot directly drive the robotic arm to perform tasks using natural language.

[0003] The rise of Large Language Models (LLMs) offers a potential solution to these challenges, possessing powerful semantic understanding and logical decomposition capabilities that can map human instructions into a sequence of steps. However, applying LLMs to robotic arms presents a "semantic implementation" problem. Existing end-to-end visual-language-action models are mostly "black box" structures, with uninterpretable reasoning and a reliance on massive amounts of labeled data and high computational resources, making it difficult to meet industrial-grade safety and real-time requirements. Furthermore, pure LLM-driven solutions lack physical common sense and spatial constraints, easily generating "illusory" actions such as illegal collisions or exceeding joint limits.

[0004] Therefore, the industry urgently needs an intelligent operation framework that integrates high-level logical reasoning with low-level physical constraints, has zero-shot learning capabilities, and is highly interpretable, to drive the transformation of robotic arms from "pre-programmed executors" to "autonomous decision-making and collaborative partners". Summary of the Invention

[0005] To address the shortcomings of existing technologies, such as the reliance on professional programming for robotic arms, the high barrier to human-computer interaction, and the difficulty in "semantic implementation" and the potential for "illusionary" actions when large language models (LLMs) are applied to robotic arms, this invention provides a training-free natural language-driven robotic arm execution method based on large language models, enabling safe and reliable intelligent interaction of robotic arms without the need for large-scale pre-training.

[0006] To achieve the above objectives, the present invention employs the following technical solution: a training-free natural language-driven robotic arm execution method based on a large language model, comprising the following steps:

[0007] Step 1: Predefine a skill atom library covering the basic operations of the robotic arm; use a speech recognition model to convert the collected human speech into text instructions; input the text instructions and preset prompt words into a large language model for semantic parsing and task reasoning to generate a skill atom execution sequence that matches the skill atom library.

[0008] Step 2: Calibrate the monocular camera to obtain camera parameters; use the monocular camera to detect the target position, and transform the spatial coordinates of the target object to the coordinate system of the robotic arm base based on the camera parameters.

[0009] Step 3: Generate the target grasping pose and realize the motion control and planning of the robotic arm.

[0010] Furthermore, the method for implementing step 1 is as follows:

[0011] Step 1.1: Construction of the skill atom library;

[0012] The skill atom library includes status inspection skills, interactive question-and-answer skills, visual detection and localization skills, and action execution skills. Among them, status inspection skills are used to obtain the current joint status and operating parameters of the robotic arm; interactive question-and-answer skills are used to control the robotic arm to perform preset posture actions in response to human interaction commands; visual detection and localization skills are used to perform target visual detection and spatial localization based on text guidance; and action execution skills are used to control the robotic arm to perform target object grasping, placement, and stacking operations.

[0013] Step 1.2: Instantiate the prompt word;

[0014] First, the role of the robotic arm assistant in the large language model is defined. The skill atom library constructed in step 1.1 is concretized into built-in functions of the robotic arm, completing the construction of the prompt context. The core content of the prompt includes: clarifying the type and calling method of the built-in functions of the robotic arm. The large language model outputs this in JSON format, where the function key corresponds to the list of built-in functions to be executed, the response key corresponds to the first-person response content, and a simple example is provided to illustrate the output specification.

[0015] Step 1.3: Speech-to-text conversion and task comprehension and arrangement;

[0016] The acquired human speech commands are converted into text commands using a speech recognition model. The text commands and predefined prompts are then input into a large language model, and task reasoning and planning are performed in conjunction with the target visual localization results to generate an action atom sequence composed of skills from the skill atom library.

[0017] Furthermore, step 2 is implemented as follows:

[0018] Step 2.1: Monocular camera calibration;

[0019] Zhang Zhengyou's calibration method was used for hand-eye calibration, providing precise parameter support for visual detection and positioning. A checkerboard calibration board with clearly identifiable corner points was prepared and placed in different orientations and positions. Images were captured from multiple perspectives using a monocular camera with the eye on the hand. Based on the OpenCV image processing library, automatic detection and precise localization of checkerboard corner points were performed on each acquired image, obtaining the pixel coordinates of the corner points in the image coordinate system. The physical dimensions of each square in the checkerboard were determined, and the Z-axis coordinate of the corner point was set to 0. All corner points were mapped to the world coordinate system, establishing a correlation between pixel coordinates and world coordinates. The intrinsic parameter matrix, distortion coefficient matrix, and homogeneous transformation matrix were calculated using the camera imaging model. The rotation matrix and translation vector of the monocular camera relative to the calibration board were integrated using the homogeneous transformation matrix.

[0020] Step 2.2: Obtaining the target's 3D coordinates guided by text;

[0021] The target description from the natural language instruction processed in step 1.3 is used as a detection prompt word and input into the YOLO-World object detection framework to perform target recognition on the scene image, outputting the 2D bounding box of the target object. Based on the FastSAM instance segmentation algorithm, instance segmentation is performed on the bounding box region to optimize the target contour, correct the accuracy of the bounding box center point position, and improve the target localization accuracy. The corrected target center point pixel coordinates are combined with the depth information provided by the depth camera and substituted into the camera intrinsic parameter matrix for inverse operation to obtain the 3D coordinates of the target point in the camera coordinate system. , , The conversion formula is:

[0022]

[0023] in, The pixel coordinates of the target center point This is the inverse of the camera intrinsic parameter matrix. Based on the eye-on-hand approach, and combined with the camera extrinsic parameters obtained from the calibration in step 2.1, the 3D coordinates of the target object in the camera coordinate system are first transformed to the coordinate system of the robotic arm's end effector. Subsequently, using the pose transformation matrix of the robotic arm's end effector relative to the base coordinate system, a second homogeneous transformation is performed on the target pose in the end effector coordinate system. Finally, the pose representation of the target object is transformed to the robotic arm's base coordinate system, completing coordinate system unification and providing accurate base coordinate parameters for subsequent robotic arm motion control and planning.

[0024] Furthermore, step 3 is implemented as follows:

[0025] First, the grasping pose is calculated. Based on the coordinate transformation results completed in step 2.2, i.e., the target object has been transformed into the three-dimensional coordinate parameters of the robot arm base coordinate system, the principal orientation of the target is estimated by combining the target contour, and the rotation angle of the target in the image plane is calculated. Based on the target's three-dimensional coordinates in the base coordinate system, the target grab point is determined. Where x, y, and z are the coordinate values ​​of the target in the coordinate system of the robotic arm base, and the end effector grasping pose matrix is ​​constructed by combining the calculated rotation angle. The expression is:

[0026]

[0027] in and The rotation matrices around the z-axis and x-axis are used to generate the end effector grasping pose parameters based on the robot arm's base coordinate system, providing precise pose support for subsequent robot arm motion control and planning. Next, inverse kinematics and trajectory planning are performed. Based on the end effector grasping pose and the target pose obtained from visual positioning, the end effector target pose matrix is ​​calculated, and the joint angles are solved using analytical or numerical iterative methods. Feasible joint commands are then selected based on joint angles, velocity, and travel limitations. If no analytical solution exists, the Jacobi pseudo-inverse iterative method is used to obtain a solution that meets the accuracy requirements within the joint constraints. The trajectory generation uses cubic polynomial interpolation to generate smooth joint motion trajectories, avoiding shocks and vibrations and ensuring stable robot arm movement.

[0028] Compared with the prior art, the present invention has at least the following beneficial effects:

[0029] 1. This invention enables zero-shot deployment without training, effectively reducing system costs. Leveraging the logical transfer capabilities of a pre-trained large language model, it eliminates the need for secondary fine-tuning and reinforcement learning training for new objects or scenes, truly achieving "out-of-the-box" functionality and significantly reducing system deployment and subsequent maintenance costs.

[0030] 2. This invention effectively avoids the risk of "illusionary" actions and improves operational safety. By combining preset prompts with inverse kinematics calculations, it achieves deep alignment between semantic understanding and physical constraints, avoiding unexecutable actions and the "illusionary" problem that occurs in large language model applications, thus ensuring the safe and feasible operation of the robotic arm.

[0031] 3. This invention offers precise positioning and strong robustness. By integrating YOLO-World's high generalization detection capability with FastSAM's fine segmentation technology, it achieves accurate target positioning, ensuring the accuracy and stability of visual positioning.

[0032] 4. This invention significantly lowers the barrier to human-computer interaction, making it suitable for non-professional users. Ordinary non-professional users do not need to master professional programming skills; they can drive the robotic arm to complete precision tasks simply through everyday speech, laying a solid technical foundation for building a human-machine collaborative working environment. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the training-free natural language driven robotic arm execution method based on a large language model according to the present invention.

[0034] Figure 2 This is a schematic diagram of the hand-eye calibration process for a monocular camera according to the present invention;

[0035] Figure 3 This is a schematic diagram illustrating the principle of obtaining target coordinates in text-guided text processing according to the present invention.

[0036] Figure 4 This is a schematic diagram of the four-segment grasping and placement process of the robotic arm of the present invention; Detailed Implementation

[0037] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0038] like Figure 1 As shown, this invention proposes a training-free natural language-driven robotic arm execution method based on a large language model. The specific implementation steps are as follows:

[0039] Step 1.1: Construction of the Skill Atom Library

[0040] The robotic arm operation instructions are abstracted into four basic atomic skills, specifically defined as follows: (1) Status check class: Define the function check_arm_state(), which is used to read the angle of each joint of the robotic arm and the opening and closing degree of the gripper, and return the status data; (2) Interactive question and answer class: Define the function arm_interact(question), which controls the robotic arm to rotate ±15° around the z-axis to achieve nodding / shaking when the input is "Has the grab been successful?", nodding is a clockwise rotation of 15° and then reset, and shaking is a counterclockwise rotation of 15° and then reset; (3) Visual detection and positioning class: Define the function visual_detect(target), which calls the YOLO-World+FastSAM model, inputs the target description, and outputs the target's three-dimensional coordinates and contour information; (4) Action execution class: Define the function grab_place(start_pos, end_pos), inputs the starting coordinates and target coordinates, and the rectangular bounding box of the control body is as follows. Figure 2 As shown, the robotic arm completes the grasping, moving, and placing actions, with the gripper closing force set to 5N and the moving speed set to 0.2m / s.

[0041] Step 1.2: Instantiate the prompt word

[0042] The skill atom library is visualized as built-in functions to complete the construction of the prompt context. Specifically: The role of the robotic arm assistant in the Qwen-Plus large language model is clearly defined, as are the types and calling methods of the built-in functions. These functions cover core functions such as posture control, gripper operation, coordinate movement, and visual processing. The large language model must output in JSON format, where the "function" key corresponds to a list of built-in functions to be executed, containing relevant parameters and arranged in the actual execution order, and the "response" key corresponds to a concise first-person response. Example output: Input "grab water bottle", output {"function":[{"name":"visual_detect","params":{"target"water bottle"}},{"name":"grab_place","params":{"start_pos":null,"end_pos":null}}],"response":"Detecting water bottle"}. The prompt words are encapsulated in JSON format to ensure that the function list output by the model corresponds to the skill atom library and that the parameter format is consistent.

[0043] Step 1.3: Speech-to-text and Task Understanding and Arrangement

[0044] The speech-to-text and task understanding and orchestration process consists of three steps: (1) Speech-to-text: The Qwen-Audio-ASR model is used to transcribe the user's spoken command "grab the water bottle on the table and put it into the box" into a text command; (2) Task orchestration: The converted text command and the preset prompt words from step 1.2 are input into the Qwen-Plus large language model to generate an action atomic sequence (JSON format): {"function":[{"name":"visual_detect","params":{"target":"water bottle"}},{"name":"visual_detect","params":{"target":"box"}},{"name":"grab_place","params":{"start_pos":(X1,Y1,Z1),"end_pos":(X2,Y2,Z2-0.05)}}],"response":"grab the water bottle and put it into the box"}.

[0045] Meanwhile, the model has logical judgment capabilities. If the input instruction contains multiple steps, it can break down the actions. If no target is detected, it outputs no action sequence and corresponding prompts, thereby ensuring the logic and feasibility of instruction parsing.

[0046] Step 2.1: Monocular camera calibration

[0047] The monocular camera calibration adopts the Zhang Zhengyou calibration method of the "eye on hand" approach. The specific operation is as follows: (1) Prepare a checkerboard calibration board with clear corner points. Its specifications are 9×12, and the side length of each square is 20mm. Fix it on the table, and the table and the robot arm base are relatively stationary; (2) Control the end of the robot arm to drive the monocular camera to take pictures of the calibration board from 20 different positions and collect 20 sets of effective pose images, such as Figure 2 As shown; (3) Based on OpenCV 4.5.5, automatically detect and accurately locate the corner points of the chessboard grid, obtain the pixel coordinates (u,v) of the corner points, map the corner points to the world coordinate system, set the Z-axis coordinate of the corner points in the world coordinate system to 0, and establish the relationship between the pixel coordinates and the world coordinates; (4) Solve the intrinsic parameter matrix K, distortion coefficient matrix D and homogeneous transformation matrix through the camera imaging model. The homogeneous transformation matrix 5) Call the cv2.undistort function to correct the original image, eliminate optical distortion, and unify the coordinate system.

[0048] Step 2.2: Obtaining the target's 3D coordinates guided by text

[0049] Text-guided acquisition of target 3D coordinates, such as Figure 3 As shown, the specific process is as follows: (1) Target detection and segmentation: The target descriptions "water bottle" and "box" after preprocessing in step 1.3 are used as detection prompt words. The YOLO-World target detection framework is input, and two-dimensional bounding boxes are output. The bounding box of the water bottle is (315,155,340,465), and the bounding box of the box is (20,160,280,420), with the unit being pixels. The FastSAM instance segmentation algorithm is called to optimize the target contour and correct the center point position. The center point coordinates of the water bottle are (160,140), and the center point coordinates of the box are (320,200). (2) Camera coordinate system coordinate calculation: The depth value of the target center point is obtained through the depth camera. The depth value of the water bottle is 0.41m, and the depth value of the box is 0.48m. The values ​​are substituted into the camera intrinsic parameter matrix for inverse operation, and the conversion formula is: The coordinates in the camera coordinate system are obtained as follows: the coordinates of the water bottle are (-0.192, -0.130, 0.410), and the coordinates of the box are (-0.151, -0.124, 0.480); (3) Coordinate system one: based on the eye-on-hand scheme, combined with the homogeneous transformation matrix The camera coordinate system coordinates are transformed to the end effector coordinate system, with the water bottle coordinates being (-0.094, 0.136, 0.348) and the box coordinates being (-0.089, 0.096, 0.418); this is then combined with the pose transformation matrix of the robotic arm's end effector relative to the base. After a second homogeneous transformation to the base coordinate system, the coordinates of the water bottle are (0.106, 0.186, 0.348) and the coordinates of the box are (0.111, 0.146, 0.418), thus completing coordinate system one.

[0050] Step 3.1: Capture Pose Calculation

[0051] The specific process of capturing pose calculation is as follows: (1) Main direction estimation: Based on the coordinates (0.106, 0.186, 0.348) of the water bottle base coordinate system obtained in step 2.2 and the target contour, calculate the rotation angle of the target in the image plane. , , Let be the principal direction vector components of the contour, where , (2) Grasping point determination: Based on the three-dimensional coordinates of the water bottle in the base coordinate system, the target grasping point is determined. (3) Pose matrix construction: combined with rotation angle Construct the end-effector grasping pose matrix , and These are the rotation matrices about the z-axis and x-axis, respectively, where the rotation matrix for a 25° rotation about the z-axis is... Rotate around the x-axis rotation matrix Multiplying the two together gives Substitute the crawling point The final end-effector grasping pose parameters are obtained as follows: X = 0.106m, Y = 0.186m, Z = 0.348m, and the attitude angles are... , , .

[0052] Step 3.2: Inverse Kinematics Solution and Trajectory Planning

[0053] The inverse kinematics solution and trajectory planning process is as follows: (1) Joint angle solution: Based on the end-effector grasping pose matrix, the target pose matrix of the end effector is calculated. The joint variable solution set is solved analytically. Combined with the joint angle and velocity constraints, the executable joint commands are selected. The angles of joints 1 to 6 are 0.15, -0.42, 0.38, -0.25, 0.32, and 0.18 rad. (2) Trajectory generation: Cubic polynomial interpolation is used. The interpolation formula is: , for , For exercise time, Generate joint trajectory within 5 seconds. , The initial joint angle, The target joint angle is set to ensure smooth, impact-free movement.

[0054] Step 3.3: Four-stage grasping and placement process

[0055] The four-stage grasping and placement process, such as Figure 4 As shown, the specific steps are as follows: (1) First stage: Move to directly above the water bottle, considering that the gripper length L is 0.1m and the safety margin is 0.1m, set the pose [0.106, 0.186, 0.548, [0, 25°], control the robotic arm to move to this position to avoid collision; (2) Second stage: adjust the gripper angle according to the rotation angle To adjust the position by 25°, align the gripper with the main direction of the water bottle; (3) Third stage: vertical gripping with the gripper. Given that the height H of the water bottle is 0.1m, adjust the z-axis coordinate to 0.368m to obtain the final gripping position [0.106, 0.186, 0.368, , 0, 25°], control the gripper to close, the closing force is 5N, and the gripping is completed; (4) Fourth stage: place the object, set the placement posture [0.111,0.146, 0.368, [0, 25°], this position is 0.05m inside the box. The robotic arm moves along the planned trajectory and opens the gripper to complete the placement.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A training-free natural language-driven robotic arm execution method based on a large language model, characterized in that, Includes the following steps: Step 1: Predefine a skill atom library covering the basic operations of the robotic arm; use a speech recognition model to convert the collected human speech into text instructions; input the text instructions and preset prompt words into a large language model for semantic parsing and task reasoning to generate a skill atom execution sequence that matches the skill atom library; Step 2: Calibrate the monocular camera to obtain camera parameters; use the monocular camera to detect the target position, and transform the spatial coordinates of the target object to the coordinate system of the robotic arm base based on the camera parameters; Step 3: Generate the target grasping pose and realize the motion control and planning of the robotic arm.

2. The method for training-free natural language-driven robotic arm execution based on a large language model as described in claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Construction of the skill atom library; The skill library includes status checking skills, interactive question-and-answer skills, visual detection and localization skills, and motion execution skills. Among them, status checking skills are used to obtain the current joint status and operating parameters of the robotic arm; interactive question-and-answer skills are used to control the robotic arm to perform preset posture actions in response to human interaction commands; visual detection and localization skills are used to perform target visual detection and spatial localization based on text guidance; and motion execution skills are used to control the robotic arm to perform target object grasping, placement, and stacking operations. Step 1.2: Instantiate the prompt word; First, the role of the robotic arm assistant in the large language model is defined, and the skill atom library constructed in step 1.1 is concretized into built-in functions of the robotic arm to complete the construction of the prompt context. The core content of the prompt includes: clarifying the type and calling method of the built-in functions of the robotic arm. The large language model outputs in JSON format, where the function key corresponds to the list of built-in functions to be executed, the response key corresponds to the first-person response content, and a simple example is provided to illustrate the output specification. Step 1.3: Speech-to-text conversion and task comprehension and arrangement; The acquired human speech commands are converted into text commands using a speech recognition model. The text commands and predefined prompts are then input into a large language model, and task reasoning and planning are performed in conjunction with the target visual localization results to generate an action atom sequence composed of skills from the skill atom library.

3. The method for training-free natural language-driven robotic arm execution based on a large language model as described in claim 1, characterized in that: Step 2 is implemented as follows: Step 2.1: Monocular camera calibration; Zhang Zhengyou's calibration method was used for hand-eye calibration, providing precise parameter support for visual detection and positioning. A checkerboard calibration board with clearly identifiable corner points was prepared and placed in different orientations and positions. Images were captured from multiple perspectives using a monocular camera with the eye on the hand. Based on the OpenCV image processing library, automatic detection and precise positioning of checkerboard corner points were performed on each acquired image to obtain the pixel coordinates of the corner points in the image coordinate system. The physical size of each square in the checkerboard was determined, and the Z-axis coordinate of the corner points was set to 0. All corner points were mapped to the world coordinate system to establish a correlation between pixel coordinates and world coordinates. The intrinsic parameter matrix, distortion coefficient matrix, and homogeneous transformation matrix were calculated using the camera imaging model. The rotation matrix and translation vector of the monocular camera relative to the calibration board were integrated using the homogeneous transformation matrix. Step 2.2: Obtaining the target's 3D coordinates guided by text; The target description in the natural language instruction processed in step 1.3 is used as a detection prompt word and input into the YOLO-World object detection framework to perform target recognition on the scene image, outputting the 2D bounding box of the target object; based on the FastSAM instance segmentation algorithm, instance segmentation is performed on the bounding box region to optimize the target contour, correct the accuracy of the bounding box center point position, and improve the target localization accuracy; the corrected target center point pixel coordinates are combined with the depth information provided by the depth camera, and substituted into the camera intrinsic parameter matrix for inverse operation to obtain the 3D coordinates of the target point in the camera coordinate system. , , The conversion formula is: ； in, The pixel coordinates of the target center point This is the inverse of the camera intrinsic parameter matrix. Based on the eye-on-hand scheme, and combined with the camera extrinsic parameters obtained from the calibration in step 2.1, the three-dimensional coordinates of the target object in the camera coordinate system are transformed to the coordinate system of the robotic arm end effector. Combined with the pose transformation matrix of the robotic arm end effector relative to the base coordinate system, a second homogeneous transformation is performed on the target pose in the end effector coordinate system. Finally, the pose representation of the target object is transformed to the robotic arm base coordinate system, completing coordinate system unification and providing accurate base coordinate parameters for subsequent robotic arm motion control and planning.

4. The method for training-free natural language-driven robotic arm execution based on a large language model as described in claim 1, characterized in that: Step 3 is implemented as follows: First, the grasping pose is calculated. Based on the coordinate transformation results completed in step 2.2, i.e., the target object has been transformed into the three-dimensional coordinate parameters of the robot arm base coordinate system, the principal direction of the target is estimated by combining the target contour, and the rotation angle of the target in the image plane is calculated. Based on the target's three-dimensional coordinates in the base coordinate system, determine the target grab point. Where x, y, and z are the coordinate values ​​of the target in the coordinate system of the robotic arm base, and the end effector grasping pose matrix is ​​constructed by combining the calculated rotation angle. The expression is: ； in and The rotation matrices around the z-axis and x-axis are respectively used to generate the end effector grasping pose parameters based on the robot arm base coordinate system, providing accurate pose support for subsequent robot arm motion control and planning. Next, inverse kinematics solution and trajectory planning are performed. Based on the end effector grasping pose and the target pose obtained by visual positioning, the target pose matrix of the end effector is calculated, and the joint angles are solved by analytical or numerical iteration methods. Then, feasible joint commands are selected based on joint angles, speed, and travel limitations. If the analytical solution does not exist, the Jacobi pseudo-inverse iteration method is used to solve it, and a solution that meets the accuracy requirements is obtained within the joint limits. The trajectory generation adopts cubic polynomial interpolation to generate smooth joint motion trajectories, avoiding impact and vibration, and ensuring the smooth movement of the robot arm.

Images