Path planning method and device for robotic arm, and storage medium

Abstract

The present invention provides a path planning method and device for a robot arm, and a storage medium, wherein the method comprises: receiving a voice grabbing instruction, wherein the voice grabbing instruction is used to instruct to grab a target object; acquiring a scene image of the location of the target object captured by a camera, and detecting the initial posture of the target object based on the scene image; determining the pre-grabbing posture of the robot arm based on the initial posture, wherein the pre-grabbing posture is the preparation posture of the robot arm before grabbing the target object; and planning the movement path of the robot arm in real time according to the pre-grabbing posture. The technical problem of low efficiency of path planning of the robot arm in the related art is solved, and the path planning efficiency of the robot arm is improved.

Description

Technical Field

[0001] The present invention relates to the field of automatic control technology, and in particular to a path planning method and device for a robotic arm, and a storage medium. Background Art

[0002] In the related art, integrating artificial intelligence into robot systems can promote human-robot collaboration, especially in complex tasks in dynamic scenarios. Traditional robot systems usually rely on pre-programmed tasks and controlled environments with limited degrees of freedom. However, in real-world tasks, especially when robots need to interact with humans in real time, robots are required to have higher adaptability and precision. Imagine that a worker needs an assistant robot to pick up a tool such as scissors, hammers, screwdrivers, etc. from the clutter in the workshop. The robot must continuously detect and grasp the tool. This simple example contains three independent tasks: natural language understanding, real-time object positioning and tracking, and real-time path planning. The task of natural language understanding enables the robot to determine the specific target; real-time object positioning and tracking is a necessary condition for obstacle avoidance and path planning; real-time path planning ensures that the robot avoids obstacles and delivers the grasped object to the target location. Some previous methods have focused on end-to-end methods based on predefined large model skill datasets. These end-to-end methods often ignore kinematic constraints, which leads to unsafe and inaccurate robot operation. These constraints and requirements are critical to meeting industrial standards, and accurate and reliable operation is of utmost importance.

[0003] In the related technology, solutions similar to human-machine collaboration use closed dictionaries, resulting in insufficient types of objects that can be recognized and limited flexibility. Most of the previous solutions use progressive path optimization methods, which have slow planning speeds and insufficient smoothness.

[0004] With respect to the above-mentioned problems existing in the related technologies, no efficient and accurate solutions have been found yet. Summary of the invention

[0005] The present invention provides a path planning method and device for a robot arm, and a storage medium to solve technical problems in related technologies.

[0006] According to one embodiment of the present invention, a path planning method for a robotic arm is provided, comprising: receiving a voice grasping instruction, wherein the voice grasping instruction is used to instruct grasping a target object; acquiring a scene image of the target object's location captured by a camera, and detecting an initial posture of the target object based on the scene image; determining a pre-grasping posture of the robotic arm based on the initial posture, wherein the pre-grasping posture is a preparation posture of the robotic arm before grasping the target object; and planning a moving path of the robotic arm in real time according to the pre-grasping posture.

[0007] Optionally, detecting the initial pose of the target object based on the scene image includes: inputting the scene image into an object recognition model, outputting a bounding box of the target object, wherein the object recognition model is an open dictionary large model; using the bounding box to generate a binary mask; and detecting the initial pose of the target object based on the binary mask.

[0008] Optionally, detecting the initial pose of the target object according to the binary mask includes: acquiring depth data of the camera; inputting the binary mask and the depth data into a pre-trained convolutional neural network, and outputting a rough pose of the target object, wherein the rough pose includes an initial position and a rough posture; and performing translation and rotation updates on the rough posture in the rough pose to obtain the corrected initial pose of the target object.

[0009] Optionally, performing translation update and rotation update on the coarse posture in the coarse pose to obtain the corrected initial posture of the target object includes: inputting the coarse posture into a pre-trained translation encoder and a rotation encoder respectively to obtain corresponding translation correction results and rotation correction results, wherein the translation encoder and the rotation encoder are both self-attention encoders; inputting the translation correction results and the rotation correction results into a posture ranking encoder to output feature embeddings of multiple posture hypotheses; linearly projecting the feature embeddings to a Kalman filter to obtain multiple score values ​​associated with the posture; and selecting the posture with the highest score value as the corrected initial posture of the target object.

[0010] Optionally, real-time planning of the movement path of the robotic arm according to the pre-grasping posture includes: determining the grasping posture of the robotic arm based on the pre-grasping posture; creating a pre-grasping path without collision constraints with the pre-grasping posture as the starting position and the grasping posture as the end position; parsing the target position of the target object based on the voice grasping instruction; creating a grasping path without collision constraints with the grasping posture as the starting position and the target position as the end position, and planning the control torque of the robotic arm in real time based on the grasping path, wherein the movement path includes the pre-grasping path and the grasping path.

[0011] Optionally, real-time planning of the control torque of the robotic arm based on the grasping path includes: constructing a rigid body dynamics model of the robotic arm; determining the constraints of the moving path, and converting the rigid body dynamics model into a state transfer equation and a target cost function based on the constraints, wherein the state transfer equation is used to solve the current position of the target object, and the target cost function is used to solve the control torque of the robotic arm to reach the next optimal path point; starting from the starting position, iteratively performing the following steps until the target object reaches the target position: obtaining the current state variables and the current control torque of the robotic arm in the current control cycle, inputting the current state variables and the current control torque into the state transfer equation and the target cost function, and outputting the target state variables and target control torque of the robotic arm in the next control cycle; judging whether the target object has reached the target position based on the target state variables, and if the target object has not reached the target position, outputting the target control torque to the robotic arm.

[0012] Optionally, the current state variable and the current control torque are input into the state transfer equation and the target cost function, and the target state variable of the robot arm in the next control cycle is output, including: using the following formula to calculate the target state variable of the robot arm in the k+1 cycle : :in, is the state variable of the k-cycle, is the control torque of k cycles, is the control torque of the k+1 cycle, N is the total number of path points of the moving path, n is the maximum number of iterations in the current cycle, k≤n, , , represents the Kronecker product, Is the size of is the identity matrix of , and h is the cycle time.

[0013] Optionally, the current state variable and the current control torque are input into a state transfer equation and a target cost function, and the target state variable and target control torque of the robotic arm in the next control cycle are output, including: minimizing the target control torque of the robotic arm in the k+1 cycle using the following target cost function: ;in, , , are the weight coefficients, is the state variable of the k-cycle, is the control torque of k cycles, is the midpoint of the path that can be reached by the camera field of view, N is the total number of path points of the moving path, and n is, is the path tracking cost function, which is used to minimize the deviation between the path point and the moving path. is a target position cost function, used to minimize the deviation between the final position of the robot arm and the target position, is an obstacle avoidance cost function, which is used to realize the collision-free constraint of the robotic arm.

[0014] According to another embodiment of the present invention, a path planning device for a robotic arm is provided, comprising: a receiving module, used to receive a voice grasping instruction, wherein the voice grasping instruction is used to instruct to grasp a target object; a detection module, used to obtain a scene image of the target object located at a position captured by a camera, and detect an initial posture of the target object based on the scene image; a determination module, used to determine a pre-grasping posture of the robotic arm based on the initial posture, wherein the pre-grasping posture is a preparation posture of the robotic arm before grasping the target object; and a planning module, used to plan the movement path of the robotic arm in real time according to the pre-grasping posture.

[0015] Optionally, the detection module includes: an output unit, used to input the scene image into an object recognition model, and output a bounding box of the target object, wherein the object recognition model is an open dictionary large model; a generation unit, used to generate a binary mask using the bounding box; and a detection unit, used to detect the initial pose of the target object based on the binary mask.

[0016] Optionally, the detection unit includes: an acquisition subunit, used to acquire the depth data of the camera; an output subunit, used to input the binary mask and the depth data into a pre-trained convolutional neural network, and output a rough pose of the target object, wherein the rough pose includes an initial position and a rough posture; and a conversion subunit, used to perform translation and rotation updates on the rough posture in the rough pose to obtain the corrected initial pose of the target object.

[0017] Optionally, the conversion subunit is also used to: input the rough posture into a pre-trained translation encoder and a rotation encoder respectively to obtain corresponding translation correction results and rotation correction results, wherein the translation encoder and the rotation encoder are both self-attention encoders; input the translation correction results and the rotation correction results into a posture ranking encoder, and output feature embeddings of multiple posture hypotheses; linearly project the feature embeddings to a Kalman filter to obtain multiple score values ​​associated with the posture; and select the posture with the highest score value as the corrected initial posture of the target object.

[0018] Optionally, the planning module includes: a determination unit, used to determine the grasping posture of the robotic arm based on the pre-grasping posture; a creation unit, used to create a pre-grasping path without collision constraints with the pre-grasping posture as the starting position and the grasping posture as the end position; a parsing unit, used to parse the target position of the target object based on the voice grasping instruction; a planning unit, used to create a grasping path without collision constraints with the grasping posture as the starting position and the target position as the end position, and to plan the control torque of the robotic arm in real time based on the grasping path, wherein the moving path includes the pre-grasping path and the grasping path.

[0019] Optionally, the planning unit includes: a construction subunit, used to construct a rigid body dynamics model of the robotic arm; a conversion subunit, used to determine the constraints of the moving path, and convert the rigid body dynamics model into a state transfer equation and a target cost function based on the constraints, wherein the state transfer equation is used to solve the current position of the target object, and the target cost function is used to solve the control torque of the robotic arm to reach the next optimal path point; an iteration subunit, used to iteratively execute the following steps starting from the starting position until the target object reaches the target position: obtain the current state variables and current control torque of the robotic arm in the current control cycle, input the current state variables and the current control torque into the state transfer equation and the target cost function, and output the target state variables and target control torque of the robotic arm in the next control cycle; judge whether the target object has reached the target position based on the target state variables, and if the target object has not reached the target position, output the target control torque to the robotic arm.

[0020] Optionally, the iterative subunit is further used to: calculate the target state variable of the robot arm in the k+1 cycle using the following formula: : ;in, is the state variable of the k-cycle, is the control torque of k cycles, is the control torque of the k+1 cycle, N is the total number of path points of the moving path, the maximum number of iterations in the current cycle, k≤n,

[0021] , , , represents the Kronecker product, Is the size of is the identity matrix of , and h is the cycle time.

[0022] Optionally, the iterative subunit is further used to: minimize the target control torque of the robotic arm in the k+1 cycle by using the following target cost function: ;in, , , are the weight coefficients, is the state variable of the k-cycle, is the control torque of k cycles, is the midpoint of the path that can be reached by the camera field of view, N is the total number of path points of the moving path, and n is, is the path tracking cost function, which is used to minimize the deviation between the path point and the moving path. is a target position cost function, used to minimize the deviation between the final position of the robot arm and the target position, is an obstacle avoidance cost function, which is used to realize the collision-free constraint of the robotic arm.

[0023] According to another aspect of an embodiment of the present application, a storage medium is further provided, which includes a stored program, and the above steps are executed when the program is run.

[0024] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the program stored in the memory.

[0025] According to yet another embodiment of the present invention, a storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned device embodiments when running.

[0026] Through the embodiments of the present invention, a voice grabbing instruction is received, wherein the voice grabbing instruction is used to instruct to grab a target object; a scene image of the location of the target object captured by a camera is obtained, and an initial posture of the target object is detected based on the scene image; a pre-grabbing posture of the robotic arm is determined based on the initial posture, target tracking and path planning are performed through the pre-grabbing posture of the target object detected and determined by the voice grabbing instruction, and the path is planned starting from the pre-grabbing posture. While ensuring constraints such as obstacle avoidance, the human-machine collaboration task is completed, thereby solving the technical problem of low efficiency of path planning of robotic arms in related technologies and improving the path planning efficiency of robotic arms. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0028] Figure 1 is a hardware structure block diagram of a robot according to an embodiment of the present invention;

[0029] Figure 2 is a flow chart of a path planning method for a robotic arm according to an embodiment of the present invention;

[0030] Figure 3 is a schematic diagram of a path for grabbing an object according to an embodiment of the present invention;

[0031] Figure 4 It is a structural block diagram of a path planning device for a robotic arm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Example 1

[0035] The method embodiment provided in the first embodiment of the present application can be executed in a robot, a mechanical arm, a controller or a similar machine control device. Taking running on a robot as an example, Figure 1 is a hardware structure diagram of a robot according to an embodiment of the present invention. Figure 1 As shown, the robot may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the robot may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the path planning device of the above-mentioned robot arm. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0036] The memory 104 can be used to store robot programs, for example, software programs and modules of application software, such as a robot program corresponding to a path planning method for a robot arm in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the robot program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the robot via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by the communication provider of the robot. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0038] In this embodiment, a path planning method for a robotic arm is provided. Figure 2 is a flow chart of a path planning method for a robotic arm according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0039] Step S202, receiving a voice grabbing instruction, wherein the voice grabbing instruction is used to instruct to grab a target object;

[0040] The voice capture instruction of this embodiment may also be a text instruction or a machine instruction.

[0041] Step S204, obtaining a scene image of the location of the target object captured by a camera, and detecting an initial position and posture of the target object based on the scene image;

[0042] The position and posture in this embodiment include the position and posture in three-dimensional coordinates.

[0043] Step S206, determining a pre-grasping posture of the robotic arm based on the initial posture, wherein the pre-grasping posture is a preparation posture of the robotic arm before grasping the target object;

[0044] Step S208: planning the movement path of the robot arm in real time according to the pre-grasping posture.

[0045] Through the above steps, a voice grabbing instruction is received, wherein the voice grabbing instruction is used to instruct to grab a target object; a scene image of the target object's location captured by a camera is acquired, and an initial posture of the target object is detected based on the scene image; a pre-grabbing posture of the robotic arm is determined based on the initial posture, and target tracking and path planning are performed on the pre-grabbing posture of the target object detected and determined by the voice grabbing instruction, and the path is planned starting from the pre-grabbing posture. While ensuring constraints such as obstacle avoidance, the human-machine collaboration task is completed, thereby solving the technical problem of low efficiency of path planning of robotic arms in related technologies and improving the path planning efficiency of robotic arms.

[0046] In an implementation scenario of the present embodiment, detecting the initial pose of the target object based on the scene image includes: inputting the scene image into an object recognition model, outputting a bounding box of the target object, wherein the object recognition model is an open dictionary large model; using the bounding box to generate a binary mask; and detecting the initial pose of the target object based on the binary mask.

[0047] Optionally, the object identification model is an OWLv2 model.

[0048] To determine the pose of a target object specified in a natural language command, such as "grab the orange drill", the key phrase is first extracted from the command, in this case "orange drill". At the same time, the RGB image is input into the OWLv2 model to generate pseudo-annotated boxes for the image-text pairs. The OWLv2 weights of the pre-trained model were used without additional training, but this method is effective and efficient in accurately locating the center of objects in the image. Compared with more complex segmentation methods that take several seconds to complete inference, the OWLv2 model can complete the localization of objects in 0.5 seconds in all experimental lengths. After obtaining the bounding box of the target object, a binary mask is generated. This mask is used as input to the subsequent object pose detection process to detect the initial pose of the target object.

[0049] In one example, detecting the initial pose of the target object according to the binary mask includes: acquiring depth data of the camera; inputting the binary mask and the depth data into a pre-trained convolutional neural network, and outputting a rough pose of the target object, wherein the rough pose includes an initial position and a rough posture; and performing translation and rotation updates on the rough posture in the rough pose to obtain the corrected initial pose of the target object.

[0050] Optionally, performing translation and rotation updates on the coarse posture in the coarse pose to obtain the corrected initial posture of the target object includes: inputting the coarse posture into a pre-trained translation encoder and a rotation encoder respectively to obtain corresponding translation correction results and rotation correction results, wherein the translation encoder and the rotation encoder are both self-attention encoders; inputting the translation correction results and the rotation correction results into a posture ranking encoder to output feature embeddings of multiple posture hypotheses; linearly projecting the feature embeddings to a Kalman filter to obtain multiple score values ​​associated with the postures; and selecting the posture with the highest score value as the corrected initial posture of the target object.

[0051] The solution of this embodiment proposes a framework for real-time object pose estimation and tracking. In the pose initialization stage, the depth sensor of the RGBD camera and the data of the RGB camera are first aligned and synchronized. Then, a rough initial pose estimate is made using the binary mask generated above to obtain a rough pose. This initial estimate is the input of the pose correction network in the subsequent stage. The architecture of the correction network is only updated in the reasoning and response object translation. and rotation update Each update is processed through a self-attention encoder and then linearly projected to the output dimension. They are expressed in terms of the camera coordinate system. and This embodiment defines the following equation to optimize the initial rough pose: :

[0052] (1)

[0053] ⊕ (2)

[0054] Among them, ⊕ means The update operation is performed on . This separation representation method effectively separates the translation from the orientation update. Using the L2 regularization function, supervised training correction network:

[0055] (3)

[0056] in, and represents the true value, and It is an empirically determined weight used to balance the loss term, and is set to 1 when applied in the present invention.

[0057] After the correction of the pose predictions, the hierarchical ranking of the pose selection stage calculates their respective scores. The network uses a hierarchical comparison and contrast validation method to calculate the loss for each pair of hypotheses and finally selects the final pose estimate with the highest score. This network uses the same backbone architecture as the refinement network for feature extraction, but is slightly modified by adding a multi-graph attention mechanism. The extracted features are concatenated, labeled, and processed through a multi-head self-attention mechanism and a pose ranking encoder to produce a feature embedding for all pose hypotheses. Then, the concatenated feature embedding Multi-head self-attention is applied to capture pose alignment information from all hypotheses. Finally, a linear projection of the features of primary interest produces a score associated with the pose estimate By selecting the pose estimation with the highest score, accurate and meaningful pose estimation can be obtained, ensuring the reliability of the final output.

[0058] To improve object pose estimation To improve accuracy and smoothness, this embodiment introduces a Kalman filter: (4)

[0059] in, is the direction matrix of the object In the following process, this embodiment uses a time interval of 100ms. and index Kalman filter

[0060] =A + (5)

[0061] Where A= , is Gaussian noise. is the Kronecker inner product (Kronecker product), is a matrix of size 3, and the measurement model is:

[0062] (6)

[0063] set up , is the normally distributed measurement noise. Then the following steps are performed:

[0064] , (7a)

[0065] , (7b)

[0066] , (7c)

[0067] , (7d)

[0068] (7e)

[0069] renew The measurement jitter in the pose selection phase is eliminated. Q and R are the user-defined process noise variance and measurement noise covariance, respectively.

[0070] In this embodiment, planning the movement path of the robot arm in real time according to the pre-grasping posture includes:

[0071] S11, determining a grasping posture of the robot arm based on the pre-grasping posture;

[0072] S12, creating a pre-grasping path without collision constraints with the pre-grasping posture as the starting position and the grasping posture as the end position;

[0073] The pre-grabbing path of this embodiment is an unloaded moving path before the robot arm starts to move the target object.

[0074] S13, parsing the target position of the target object based on the voice capture instruction;

[0075] S14, creating a grasping path without collision constraints with the grasping posture as the starting position and the target position as the end position, and planning the control torque of the robotic arm in real time based on the grasping path, wherein the moving path includes the pre-grasping path and the grasping path.

[0076] In one example, real-time planning of the control torque of the robotic arm based on the grasping path includes: constructing a rigid body dynamics model of the robotic arm; determining the constraints of the moving path, and converting the rigid body dynamics model into a state transfer equation and a target cost function based on the constraints, wherein the state transfer equation is used to solve the current position of the target object, and the target cost function is used to solve the control torque of the robotic arm to reach the next optimal path point; starting from the starting position, iteratively performing the following steps until the target object reaches the target position: obtaining the current state variables and the current control torque of the robotic arm in the current control cycle, inputting the current state variables and the current control torque into the state transfer equation and the target cost function, and outputting the target state variables and target control torque of the robotic arm in the next control cycle; judging whether the target object has reached the target position based on the target state variables, and outputting the target control torque to the robotic arm if the target object has not reached the target position.

[0077] The rigid body dynamics model of this embodiment is a rigid body dynamics model of a fully driven robot, and the expression is:

[0078] , (8)

[0079] Using generalized coordinates and the positive definite mass matrix , Coriolis matrix and centrifugal force terms , the gravitational potential energy term and generalized moment The Cartesian end-effector pose of a robot (robotic arm) obtained by forward dynamics can be described as a homogeneous change and is given by the following formula:

[0080] (9)

[0081] Rotation Matrix Contains pose , where 0 is a zero matrix of matching size.

[0082] The derived RH-TrajOpt requires a target and center point in joint space. These joint space solutions are solved from the object pose estimation using analytical inverse kinematics. Compared with numerical methods, analytical solutions are much more efficient, and this speed increase is very important for practical applications.

[0083] The solution of this embodiment uses the calculated torque state feedback to compensate for the nonlinear robot dynamics:

[0084] , (10)

[0085] This formula simplifies the dynamic model (8) to a double integrator system: (11)

[0086] Using the new control input , the rest of the linear dynamics system (11) is stable using the following formula:

[0087] (12)

[0088] in, and is the gain matrix that exponentially stabilizes the closed-loop path error dynamics. The desired reference path in joint space is given by and its time derivative and And the time t is given.

[0089] Optionally, the current state variable and the current control torque are input into the state transfer equation and the target cost function, and the target state variable of the robot arm in the next control cycle is output, including: using the following formula to calculate the target state variable of the robot arm in the k+1 cycle : ;in, is the state variable of the k-cycle, is the control torque of k cycles, is the control torque of the k+1 cycle, N is the total number of path points of the moving path, n is the maximum number of iterations in the current cycle, k≤n, , , represents the Kronecker product, Is the size of is the identity matrix of , and h is the cycle time.

[0090] Among them, n is the preset maximum number of iterations. At each path point, one iteration is performed to obtain the optimal state variable of the current path point.

[0091] Optionally, the current state variable and the current control torque are input into a state transfer equation and a target cost function, and the target state variable and target control torque of the robotic arm in the next control cycle are output, including: minimizing the target control torque of the robotic arm in the k+1 cycle using the following target cost function: ;in, , , are the weight coefficients, is the state variable of the k-cycle, is the control torque of k cycles, is the midpoint of the path that can be reached by the camera field of view, N is the total number of path points of the moving path, and n is, is the path tracking cost function, which is used to minimize the deviation between the path point and the moving path. is a target position cost function, used to minimize the deviation between the final position of the robot arm and the target position, is an obstacle avoidance cost function, which is used to realize the collision-free constraint of the robotic arm.

[0092] The planner in this embodiment is used to plan the optimal path points, and the controller is used to control the control torque of the robot arm. The controller requires at least two consecutive differentiable reference paths from the planner. Triple integral dynamics is applied to the planning algorithm to improve the path smoothness. The state vector is defined as ,enter , assuming the sampling time is , the piecewise linear input is , we can get the first-order discrete-time state space formula:

[0093] (13)

[0094] in

[0095] , (14)

[0096] symbol, represents the Kronecker product, Is the size of The scheme of this embodiment calculates the inverse kinematics solutions of the pre-grasp and grasp postures and the positioning and positioning postures respectively to obtain the midpoint configuration of the joint space and target configuration In order to ensure that the target position is reached along the path, once the midpoint position It is reachable within the field of vision, and the planning scope is divided into two parts, namely and 1. If the path point is reachable, the planning algorithm introduces a constraint to make it pass within a certain tolerance range. For nodes that are reachable within the view, the planner adds an endpoint constraint.

[0097] The optimal system state for the planner's nth iteration and input The following cost function is obtained from the discrete-time optimization problem:

[0098]

[0099]

[0100] The constraints are as follows:

[0101] (15b) (15c)

[0102] (15d)

[0103] , (15e)

[0104] (15f)

[0105] Among them, the cost coefficient and Drive the path to the path point and the target position respectively, the path tracking cost is: the cost coefficient Drive, aiming to minimize the deviation between the robot path point and the planned path. Target position cost: determined by the cost coefficient The driving aim is to minimize the deviation between the robot’s final position and the target position. It is designed to avoid obstacles and ensure the safety and feasibility of the path. The regularization cost ensures the path is smooth. is an obstacle avoidance term. Weight , , is an adjustable parameter. The path must follow the system dynamics (15b), where (15b) is the state variable The update formula is , whose initial state is obtained from the previous planner iteration (15c). The path midpoint is constrained to the equilibrium point in (15d), and the upper limit is used , and lower limit , To limit the input and output of the state, (15e) represents the upper and lower limit constraints of the state. The midpoints of the split time domain are constrained to the path point tolerance set and target tolerance set In , (15f) represents the path midpoint and target position constraints.

[0106] The optimized system state and control input are solved by the above state transfer equations and cost functions to ensure that the robot can reach the target position along a smooth and achievable path while meeting various physical constraints and tolerance requirements.

[0107] The solution of this embodiment includes three functional modules: language-based object detection, real-time object pose estimation, and real-time path optimization. In a workshop operation scenario, the intelligent robot grabs the tool with a known CAD model and places it at the target location. First, the semantically driven object detection module processes the image to determine the two-dimensional position of the tool and generates a binary mask. The mask is combined with the CAD model, and the real-time object pose positioning module estimates the initial pose of the tool, which is then smoothed by the Kalman filter to improve the pose of the object. Finally, the real-time trajectory optimization module plans and executes the grasping operation and obstacle avoidance to ensure smooth and collision-free operation.

[0108] The solution of this embodiment proposes a modular zero-shot framework, which enables the robot to realize open dictionary understanding of human language through a large prediction model, and then realize open dictionary understanding of target objects through a large image model, and then put into path planning and tracking modules for human-machine collaboration. A path planning method for human-machine collaboration performs target tracking and path planning based on the output of the framework pre-order model, ensuring obstacle avoidance while completing human-machine collaboration tasks.

[0109] The present invention uses a state machine to guide the robot to complete the grasping and obstacle avoidance tasks. First, the robot must approach the object to grasp it. The solution of this embodiment derives the grasping posture and pre-grasping path from the posture of the object. The pre-grasping posture is located above the object to ensure the correct approach direction and collision-free grasping. Figure 3 It is a schematic diagram of the path for grasping an object in an embodiment of the present invention. If the object moves beyond a tolerance, the pre-grasping posture will change dynamically. When the robot end effector reaches the grasping position, the robot grasps the object and moves according to the orientation planning. The pre-grasping posture is similar.

[0110] In the process of human-machine collaboration, the closed dictionary method used in the past limited the flexibility of the robot's activities. The present invention uses an open dictionary based on large model technology, which effectively improves the flexibility of the robot's activities. At the same time, the path planning method is integrated into the entire framework, so that the robot can perform efficient path planning and meet obstacle avoidance requirements at the same time.

[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0112] Example 2

[0113] In the present embodiment, a device is also provided, and the device is used to implement the above-mentioned embodiment and preferred implementation mode, and the description has been made no further. As used below, the term "module" can implement the combination of software and hardware of the reservation function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or the combination of software and hardware, can also be conceived.

[0114] Figure 4 is a structural block diagram of a path planning device for a robotic arm according to an embodiment of the present invention. Figure 4 As shown, including:

[0115] The receiving module 40 is used to receive a voice grabbing instruction, wherein the voice grabbing instruction is used to instruct to grab a target object;

[0116] A detection module 42, configured to obtain a scene image of the location of the target object captured by a camera, and detect an initial position and posture of the target object based on the scene image;

[0117] A determination module 44 is used to determine a pre-grasping posture of the robotic arm based on the initial posture, wherein the pre-grasping posture is a preparation posture of the robotic arm before grasping the target object;

[0118] The planning module 46 is used to plan the movement path of the robot arm in real time according to the pre-grasping posture.

[0119] Optionally, the detection module includes: an output unit, used to input the scene image into an object recognition model, and output a bounding box of the target object, wherein the object recognition model is an open dictionary large model; a generation unit, used to generate a binary mask using the bounding box; and a detection unit, used to detect the initial pose of the target object based on the binary mask.

[0120] Optionally, the detection unit includes: an acquisition subunit, used to acquire the depth data of the camera; an output subunit, used to input the binary mask and the depth data into a pre-trained convolutional neural network, and output a rough pose of the target object, wherein the rough pose includes an initial position and a rough posture; and a conversion subunit, used to perform translation and rotation updates on the rough posture in the rough pose to obtain the corrected initial pose of the target object.

[0121] Optionally, the conversion subunit is also used to: input the rough posture into a pre-trained translation encoder and a rotation encoder respectively to obtain corresponding translation correction results and rotation correction results, wherein the translation encoder and the rotation encoder are both self-attention encoders; input the translation correction results and the rotation correction results into a posture ranking encoder, and output feature embeddings of multiple posture hypotheses; linearly project the feature embeddings to a Kalman filter to obtain multiple score values ​​associated with the posture; and select the posture with the highest score value as the corrected initial posture of the target object.

[0122] Optionally, the planning module includes: a determination unit, used to determine the grasping posture of the robotic arm based on the pre-grasping posture; a creation unit, used to create a pre-grasping path without collision constraints with the pre-grasping posture as the starting position and the grasping posture as the end position; a parsing unit, used to parse the target position of the target object based on the voice grasping instruction; a planning unit, used to create a grasping path without collision constraints with the grasping posture as the starting position and the target position as the end position, and to plan the control torque of the robotic arm in real time based on the grasping path, wherein the moving path includes the pre-grasping path and the grasping path.

[0123] Optionally, the planning unit includes: a construction subunit, used to construct a rigid body dynamics model of the robotic arm; a conversion subunit, used to determine the constraints of the moving path, and convert the rigid body dynamics model into a state transfer equation and a target cost function based on the constraints, wherein the state transfer equation is used to solve the current position of the target object, and the target cost function is used to solve the control torque of the robotic arm to reach the next optimal path point; an iteration subunit, used to iteratively execute the following steps starting from the starting position until the target object reaches the target position: obtain the current state variables and current control torque of the robotic arm in the current control cycle, input the current state variables and the current control torque into the state transfer equation and the target cost function, and output the target state variables and target control torque of the robotic arm in the next control cycle; judge whether the target object has reached the target position based on the target state variables, and if the target object has not reached the target position, output the target control torque to the robotic arm.

[0124] Optionally, the iterative subunit is further used to: calculate the target state variable of the robot arm in the k+1 cycle using the following formula: : ;in, is the state variable of the k-cycle, is the control torque of k cycles, is the control torque of the k+1 cycle, N is the total number of path points of the moving path, n is the maximum number of iterations in the current cycle, k≤n, , , represents the Kronecker product, Is the size of is the identity matrix of , and h is the cycle time.

[0125] Optionally, the iterative subunit is further used to: minimize the target control torque of the robotic arm in the k+1 cycle by using the following target cost function: ;in, , , are the weight coefficients, is the state variable of the k-cycle, is the control torque of k cycles, is the midpoint of the path that can be reached by the camera field of view, N is the total number of path points of the moving path, and n is, is the path tracking cost function, which is used to minimize the deviation between the path point and the moving path. is a target position cost function, used to minimize the deviation between the final position of the robot arm and the target position, is an obstacle avoidance cost function, which is used to realize the collision-free constraint of the robotic arm.

[0126] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0127] Example 3

[0128] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0129] Optionally, in this embodiment, the above storage medium may be configured to store a computer program for executing:

[0130] S1, receiving a voice grabbing instruction, wherein the voice grabbing instruction is used to instruct to grab a target object;

[0131] S2, acquiring a scene image of the location of the target object captured by a camera, and detecting an initial position and posture of the target object based on the scene image;

[0132] S3, determining a pre-grasping posture of the robotic arm based on the initial posture, wherein the pre-grasping posture is a preparation posture of the robotic arm before grasping the target object;

[0133] S4, planning the moving path of the robot arm in real time according to the pre-grasping posture.

[0134] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0135] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0136] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0137] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0138] S1, receiving a voice grabbing instruction, wherein the voice grabbing instruction is used to instruct to grab a target object;

[0139] S2, acquiring a scene image of the location of the target object captured by a camera, and detecting an initial position and posture of the target object based on the scene image;

[0140] S3, determining a pre-grasping posture of the robotic arm based on the initial posture, wherein the pre-grasping posture is a preparation posture of the robotic arm before grasping the target object;

[0141] S4, planning the moving path of the robot arm in real time according to the pre-grasping posture.

[0142] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0143] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0144] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0148] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, controller or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., various media that can store program codes.

[0149] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A path planning method for a robotic arm, characterized in that: include: Receiving a voice grabbing instruction, wherein the voice grabbing instruction is used to instruct to grab a target object; Acquire a scene image of the location of the target object captured by a camera, and detect an initial position and posture of the target object based on the scene image; Determining a pre-grasping posture of the robotic arm based on the initial posture, wherein the pre-grasping posture is a preparation posture of the robotic arm before grasping the target object; Planning the movement path of the robot arm in real time according to the pre-grasping posture; Wherein, detecting the initial pose of the target object based on the scene image includes: inputting the scene image into an object recognition model, and outputting a bounding box of the target object, wherein the object recognition model is an open dictionary large model; using the bounding box to generate a binary mask; detecting the initial pose of the target object according to the binary mask, wherein detecting the initial pose of the target object according to the binary mask includes: acquiring depth data of the camera; inputting the binary mask and the depth data into a pre-trained convolutional neural network, and outputting a rough pose of the target object, wherein the rough pose includes an initial position and a rough pose; performing translation update and rotation update on the rough pose in the rough pose to obtain the corrected initial pose of the target object.

2. The method according to claim 1, characterized in that The rough pose in the rough pose is updated by translation and rotation to obtain the corrected initial pose of the target object, which includes: Inputting the rough posture into a pre-trained translation encoder and a rotation encoder respectively to obtain corresponding translation correction results and rotation correction results, wherein the translation encoder and the rotation encoder are both self-attention encoders; Inputting the translation correction result and the rotation correction result into a posture ranking encoder, and outputting feature embeddings of multiple posture hypotheses; Embedding the feature into a linear projection to a Kalman filter to obtain a plurality of score values ​​associated with the posture; The posture with the highest score is selected as the corrected initial posture of the target object.

3. The method according to claim 1, characterized in that Planning the movement path of the robot arm in real time according to the pre-grasping posture includes: Determining a grasping posture of the robotic arm based on the pre-grasping posture; A pre-grasping path without collision constraints is created with the pre-grasping posture as the starting position and the grasping posture as the end position; parsing a target position of the target object based on the voice grabbing instruction; A grasping path without collision constraints is created with the grasping posture as the starting position and the target position as the end position, and the control torque of the robotic arm is planned in real time based on the grasping path, wherein the moving path includes the pre-grasping path and the grasping path.

4. The method according to claim 3, characterized in that: Real-time planning of the control torque of the robotic arm based on the grasping path includes: Constructing a rigid body dynamics model of the robotic arm; Determine the constraint conditions of the moving path, and convert the rigid body dynamics model into a state transfer equation and a target cost function based on the constraint conditions, wherein the state transfer equation is used to solve the current position of the target object, and the target cost function is used to solve the control torque of the robot arm to reach the next optimal path point; Starting from the starting position, iteratively perform the following steps until the target object reaches the target position: obtain the current state variable and the current control torque of the robotic arm in the current control cycle, input the current state variable and the current control torque into the state transfer equation and the target cost function, and output the target state variable and target control torque of the robotic arm in the next control cycle; based on the target state variable, determine whether the target object has reached the target position, and if the target object has not reached the target position, output the target control torque to the robotic arm.

5. The method according to claim 4, characterized in that Inputting the current state variable and the current control torque into the state transfer equation and the target cost function, and outputting the target state variable of the robot arm in the next control cycle, includes: The target state variable of the robot arm in the k+1 cycle is calculated using the following formula: : ； in, is the state variable of the k-cycle, is the control torque of k cycles, is the control torque of the k+1 cycle, n is the maximum number of iterations in the current cycle, k≤n, ， ， represents the Kronecker product, Is the size of is the identity matrix of , and h is the cycle time.

6. The method according to claim 4, characterized in that Inputting the current state variable and the current control torque into the state transfer equation and the target cost function, and outputting the target state variable and the target control torque of the robot arm in the next control cycle, including: The target control torque of the robot arm in the k+1 cycle is calculated by minimizing the following target cost function: ； in, , , are the weight coefficients, is the state variable of the k-cycle, is the control torque of k cycles, is the midpoint of the path that can be reached by the camera field of view, N is the total number of path points of the moving path, n is the maximum number of iterations in the current cycle, is the path tracking cost function, which is used to minimize the deviation between the path point and the moving path. is a target position cost function, used to minimize the deviation between the final position of the robot arm and the target position, is an obstacle avoidance cost function, which is used to realize the collision-free constraint of the robotic arm.

7. A path planning device for a robotic arm, characterized in that: include: A receiving module, used for receiving a voice grabbing instruction, wherein the voice grabbing instruction is used for instructing to grab a target object; A detection module, used to obtain a scene image of the location of the target object captured by a camera, and detect an initial position and posture of the target object based on the scene image; A determination module, configured to determine a pre-grasping posture of the robotic arm based on the initial posture, wherein the pre-grasping posture is a preparation posture of the robotic arm before grasping the target object; A planning module, used for planning the movement path of the robot arm in real time according to the pre-grasping posture; The detection module includes: an output unit, which is used to input the scene image into an object recognition model and output the bounding box of the target object, wherein the object recognition model is an open dictionary large model; a generation unit, which is used to generate a binary mask using the bounding box; a detection unit, which is used to detect the initial pose of the target object according to the binary mask, wherein the detection unit includes: an acquisition subunit, which is used to acquire the depth data of the camera; an output subunit, which is used to input the binary mask and the depth data into a pre-trained convolutional neural network, and output the rough pose of the target object, wherein the rough pose includes an initial position and a rough pose; a conversion subunit, which is used to perform translation and rotation updates on the rough pose in the rough pose to obtain the corrected initial pose of the target object.

8. A storage medium, characterized in that: A computer program is stored in the storage medium, wherein the computer program is configured to execute the steps of the path planning method for a robotic arm described in any one of claims 1 to 6 when running.

Images