Mechanical arm grabbing method and system based on dynamic visual servo

Abstract

The invention relates to a mechanical arm grabbing method and system based on dynamic visual servo, and belongs to the technical field of automatic control. The method comprises the following steps: acquiring an initial depth image of a target object acquired by a depth camera; determining the shape and position of the target object based on the initial depth image, and performing feature extraction on the initial depth image based on a set feature extraction network to obtain initial key feature points of the target object; the shape and the position of the target object and the initial key feature points of the target object are input into a set track model, and an initial grabbing path of the mechanical arm is determined; and the mechanical arm is controlled to grab the target object according to the initial grabbing path, and in the grabbing process, the initial grabbing path is dynamically adjusted for grabbing. In this way, the grabbing success rate can be increased, and the risk that the target object is damaged is reduced.

Description

Technical Field

[0001] The present invention relates to the field of automatic control technology, and in particular to a mechanical arm grasping method and system based on dynamic visual servoing. Background Art

[0002] In the field of modern industrial automation and intelligent manufacturing, robotic arms are increasingly used, especially in industries such as precision manufacturing, logistics sorting, medical surgery, and service robots.

[0003] Research has found that most traditional robotic arm systems rely on preset programs and fixed operating modes, which limits their flexibility and adaptability in handling irregular objects or easily deformed objects. Irregular objects or easily deformed objects pose great difficulties to the grasping operation of the robotic arm due to the diversity of their shapes, sizes and materials. For example, for irregular objects, it is difficult to grasp them effectively and accurately according to the conventional preset procedures, and when the irregular objects are fragile objects, they are easily damaged. For another example, for easily deformed objects, if they are grasped according to the conventional preset procedures, it is easy to cause the object to be excessively deformed and unstable, or cause damage to the easily deformed objects. Summary of the invention

[0004] In order to solve the above-mentioned problems in the prior art, the present invention provides a robot arm grasping method and system based on dynamic visual servoing.

[0005] In a first aspect, an embodiment of the present application provides a robotic arm grasping method based on dynamic visual servoing, comprising: obtaining an initial depth image of a target object acquired by a depth camera; determining the shape and position of the target object based on the initial depth image, and performing feature extraction on the initial depth image based on a set feature extraction network to acquire initial key feature points of the target object; inputting the shape and position of the target object, and the initial key feature points of the target object into a set trajectory model to determine an initial grasping path of the robotic arm; controlling the robotic arm to grasp the target object with the initial grasping path, and dynamically adjusting the initial grasping path for grasping during the grasping process; wherein dynamically adjusting the initial grasping path comprises: continuously acquiring a process depth image of the target object acquired by the depth camera, and performing feature extraction on the process depth image based on the set feature extraction network to acquire process key feature points of the target object; and adjusting the initial grasping path based on the process key feature points.

[0006] Optionally, adjusting the initial grasping path based on the process key feature points includes: calculating the deviation between the current position of the robot arm and the ideal grasping position based on the process key feature points and preset grasping features of the target object; wherein the ideal grasping position corresponds to the preset grasping features of the target object; the preset grasping features of the target object are determined based on the initial key feature points of the target object, or are pre-stored grasping features set by the user; and dynamically adjusting the initial grasping path based on the deviation.

[0007] Optionally, the set feature extraction network includes: a combination of a convolutional neural network sub-model and a recursive neural network; wherein the convolutional neural network sub-model is used to extract features of depth images and obtain key feature points of the target object; the recursive neural network is used to receive key feature points of depth images of different time frames and track features of depth images of different time frames; or, the set feature extraction network includes: a combination of a convolutional neural network sub-model and a long short-term memory network; wherein the convolutional neural network sub-model is used to extract features of depth images and obtain key feature points of the target object; the long short-term memory network is used to receive key feature points of depth images of different time frames and track features of depth images of different time frames.

[0008] Optionally, the set feature extraction network is also used to classify and mark newly appeared key feature points, and to relocate or remove disappeared or moved key feature points.

[0009] Optionally, the method further includes: determining a deformation degree of the target object based on key feature points corresponding to depth images of different frames; and controlling a grasping force of the robotic arm based on the deformation degree of the target object.

[0010] Optionally, the set trajectory model is generated by training through a reinforcement learning algorithm; the set trajectory model defines a reward function; the reward function is configured as: when the action of the robotic arm makes it closer to the sample object than the previous action, or the grasping position does not cause a deformation degree greater than a set threshold to the sample object, or the grasping position does not cause damage to the sample object, a positive reward is given; when the action of the robotic arm makes it farther away from the sample object than the previous action, or the grasping position causes a deformation degree greater than the set threshold to the sample object, or the grasping position causes damage to the sample object, a negative reward is given.

[0011] Optionally, a plurality of tactile sensors are installed on the end effector of the robotic arm; the method further includes: receiving detection pressure of the plurality of tactile sensors; when the detection pressure of the plurality of tactile sensors is greater than a first set pressure threshold, reducing the gripping force of the end effector of the robotic arm; when the detection pressure of the plurality of tactile sensors is less than a second set pressure threshold, increasing the gripping force of the end effector of the robotic arm; wherein the second set pressure threshold is less than the first set pressure threshold.

[0012] Optionally, the method also includes: acquiring spectral information of the target object collected by a spectral sensor; determining the material of the target object by combining the initial depth image and the spectral information; and determining the posture of the end effector of the robotic arm based on the material of the target object and the shape of the target object.

[0013] Optionally, multiple tactile sensors are installed on the end effector of the robotic arm; the method also includes: determining a pressure threshold that matches the target object based on the material of the target object; receiving the detection pressure of the multiple tactile sensors; and dynamically adjusting the gripping force of the end effector of the robotic arm based on the detection pressure of the multiple tactile sensors and the pressure threshold that matches the target object.

[0014] In a second aspect, an embodiment of the present application provides a dynamic vision-based robotic arm grasping system, comprising: a first acquisition module, used to acquire an initial depth image of a target object acquired by a depth camera; a second acquisition module, used to determine the shape and position of the target object based on the initial depth image, and to extract features from the initial depth image based on a set feature extraction network to acquire initial key feature points of the target object; a path determination module, used to input the shape and position of the target object, and the initial key feature points of the target object into a set trajectory model to determine the initial grasping path of the robotic arm; a grasping module, used to control the robotic arm to grasp the target object with the initial grasping path, and to dynamically adjust the initial grasping path for grasping during the grasping process; wherein, dynamically adjusting the initial grasping path comprises: continuously acquiring a process depth image of the target object acquired by the depth camera, and to extract features from the process depth image based on the set feature extraction network to acquire process key feature points of the target object; and adjusting the initial grasping path based on the process key feature points.

[0015] The beneficial effects of the present invention include: First, in this application, the initial depth image is feature extracted based on the set feature extraction network to obtain the initial key feature points of the target object, and then the shape and position of the target object and the initial key feature points of the target object are input into the set trajectory model to determine the initial grasping path of the robot arm. That is, for each target object, it involves the acquisition of the depth image, the extraction of the initial key feature points, and the generation of the path based on the shape and position of the target object and the initial key feature points of the target object. It can be seen that this method can effectively and targetedly plan the grasping of different target objects, especially for the grasping of irregular objects, which can greatly improve the accuracy and effectiveness of grasping compared to the traditional grasping only through preset programs or fixed modes. And this processing method can be more conveniently applied to the scene of object switching. For example, the current robot arm is grasping a large number of objects A, but due to changes in the process, it is necessary to start grasping a large number of objects B, and this method can seamlessly connect the grasping switching process of the two objects without any human intervention.

[0016] Second, considering that in the actual grasping process, there may be factors involving the object itself that may lead to situations where high-precision grasping cannot be performed, such as deformation of the deformable object itself, or deformation during grasping that causes changes in the ideal grasping point, which in turn affects the grasping success rate according to the initial grasping path. Therefore, in this application, the changes in the ideal grasping point are analyzed in combination with the key feature points of the process to adjust the initial grasping path. For example, in the grasping process of irregular objects, the initial grasping path is further optimized based on the key feature points of the process to avoid damage to the irregular objects. That is, during the grasping process, the process depth image is continuously acquired, and the key feature points are updated to dynamically adjust the grasping path. Through this real-time feedback mechanism, the high-precision grasping of the robot arm is guaranteed, so that the above grasping process can adapt to the slight movement or deformation of the target object, thereby improving the success rate of grasping and reducing the risk of damage to the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of the steps of a first mechanical arm grasping method based on dynamic visual servoing provided by an embodiment of the present invention; Figure 2 A flowchart of the steps of a second mechanical arm grasping method based on dynamic visual servoing provided by an embodiment of the present invention; Figure 3 A module block diagram of a first mechanical arm grasping system based on dynamic visual servoing provided by an embodiment of the present invention; Figure 4 A module block diagram of a second mechanical arm grasping system based on dynamic visual servoing provided by an embodiment of the present invention; Figure 5A module block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0019] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0020] Research has found that most traditional robotic arm systems rely on preset programs and fixed operating modes, which limits their flexibility and adaptability in handling irregular objects or easily deformed objects. Irregular objects or easily deformed objects pose great difficulties to the grasping operation of the robotic arm due to the diversity of their shapes, sizes and materials. For example, for irregular objects, it is difficult to grasp them effectively and accurately according to the conventional preset procedures, and when the irregular objects are fragile objects, they are easily damaged. For another example, for easily deformed objects, if they are grasped according to the conventional preset procedures, it is easy to cause the object to be excessively deformed and unstable, or cause damage to the easily deformed objects.

[0021] In view of the above problems, the present application proposes the following embodiments to solve the above technical problems.

[0022] See also Figure 1 , an embodiment of the present application provides a robotic arm grasping method based on dynamic visual servoing, including: steps 101 to 104.

[0023] Step 101: Acquire an initial depth image of a target object captured by a depth camera.

[0024] First, the depth camera can be aimed at the target object to be grasped and photograph it to obtain an initial depth image of the target object in a static state.

[0025] It should be noted that this process occurs before the robotic arm grasps the target object, and the target object is initially captured by the depth camera.

[0026] In addition, after the initial depth image is acquired, the initial depth image may be preprocessed, including denoising, grayscale conversion and other operations, to improve the image quality for subsequent processing.

[0027] Step 102: Based on the initial depth image, determine the shape and position of the target object, and extract features from the initial depth image based on a set feature extraction network to obtain initial key feature points of the target object.

[0028] This step involves two parts. The first part is to use the initial depth image to determine the shape and position of the target object. For example, edge detection is used to find the boundary of the target object on the depth image, and then the shape of the target object is determined in turn. The position of the target object can be directly determined by analyzing the pixel values ​​in the depth image to determine the position of the target object in three-dimensional space.

[0029] The second part is to extract features from the initial depth image based on the pre-trained feature extraction network, and then obtain the initial key feature points of the target object. It should be noted that the initial key feature points of the target object can correspond to features such as the edge and texture of the target object.

[0030] Step 103: Input the shape and position of the target object and the initial key feature points of the target object into the set trajectory model to determine the initial grasping path of the robot arm.

[0031] Then, the initial grasping path of the robot arm is formulated based on the shape and position of the target object and the initial key feature points of the target object.

[0032] In the embodiment of the present application, the initial grasping path of the robot arm is determined by a pre-trained set trajectory model.

[0033] Specifically, the input data of the set trajectory model includes three items of data: the shape and position of the target object, and the initial key feature points of the target object. The output of the set trajectory model is the initial grasping path of the robot arm.

[0034] In the embodiment of the present application, the robotic arm may be a flexible robotic arm.

[0035] In addition, it should be noted that the above-mentioned initial grasping path includes two parts. The first part is the process of the robot arm moving to the location of the target object, and the second part is the process of the robot arm grasping the target object.

[0036] Step 104: Control the robot arm to grasp the target object using the initial grasping path, and during the grasping process, dynamically adjust the initial grasping path to grasp the object.

[0037] After the initial grasping path is determined, motion control instructions are generated to control the robot arm to grasp the target object along the initial grasping path. In addition, dynamic optimization is performed during the grasping process. For details, please refer to Figure 2, wherein dynamically adjusting the initial crawling path includes: step 201 to step 202.

[0038] Step 201: continuously acquiring a process depth image of a target object captured by a depth camera, and extracting features from the process depth image based on a set feature extraction network to obtain process key feature points of the target object.

[0039] It should be noted that the feature extraction network set in this step is the same as the feature extraction network set in step 102. Therefore, the same parts can be referenced to each other. In this process, only the depth image and the key feature points are distinguished. During the initial acquisition and feature extraction, the depth image is an initial depth image, which corresponds to the initial key feature points. During the capture process, the depth image is a process depth image, which corresponds to the process key feature points.

[0040] Step 202: Adjust the initial grasping path based on the key feature points of the process.

[0041] Finally, the initial grasping path is dynamically adjusted according to the updated key feature points. It should be noted that, for irregular objects or deformable objects, an initial grasping path can be provided in the aforementioned steps, but in the actual grasping process, factors of the object itself may lead to the inability to grasp with high precision, such as deformation of the deformable object itself, or deformation during grasping, which causes the ideal grasping point position to change, thereby affecting the grasping success rate according to the initial grasping path. Therefore, it is necessary to analyze the changes in the ideal grasping point position in combination with the key feature points of the process to adjust the initial grasping path. For another example, for irregular objects, the grasping method generated only by the initial key feature points may have low accuracy. Therefore, in the process of grasping irregular objects, the initial grasping path can be further optimized based on the key feature points of the process to avoid damage to the irregular objects.

[0042] In summary, the robot arm grasping method based on dynamic visual servoing provided in the embodiment of the present application has the following beneficial effects: First, in this application, the initial depth image is feature extracted based on the set feature extraction network to obtain the initial key feature points of the target object, and then the shape and position of the target object and the initial key feature points of the target object are input into the set trajectory model to determine the initial grasping path of the robot arm. That is, for each target object, it involves the acquisition of the depth image, the extraction of the initial key feature points, and the generation of the path based on the shape and position of the target object and the initial key feature points of the target object. It can be seen that this method can effectively and targetedly plan the grasping of different target objects, especially for the grasping of irregular objects, which can greatly improve the accuracy and effectiveness of grasping compared to the traditional grasping only through preset programs or fixed modes. And this processing method can be more conveniently applied to the scene of object switching. For example, the current robot arm is grasping a large number of objects A, but due to changes in the process, it is necessary to start grasping a large number of objects B, and this method can seamlessly connect the grasping switching process of the two objects without any human intervention.

[0043] Second, considering that in the actual grasping process, there may be factors involving the object itself that may lead to situations where high-precision grasping cannot be performed, such as deformation of the deformable object itself, or deformation during grasping that causes changes in the ideal grasping point, which in turn affects the grasping success rate according to the initial grasping path. Therefore, in this application, the changes in the ideal grasping point are analyzed in combination with the key feature points of the process to adjust the initial grasping path. For example, in the grasping process of irregular objects, the initial grasping path is further optimized based on the key feature points of the process to avoid damage to the irregular objects. That is, during the grasping process, the process depth image is continuously acquired, and the key feature points are updated to dynamically adjust the grasping path. Through this real-time feedback mechanism, the high-precision grasping of the robot arm is guaranteed, so that the above grasping process can adapt to the slight movement or deformation of the target object, thereby improving the success rate of grasping and reducing the risk of damage to the target object.

[0044] Optionally, the above steps adjust the initial grasping path based on the key feature points of the process, including: calculating the deviation between the current position of the robot arm and the ideal grasping position based on the key feature points of the process and preset grasping features of the target object; and dynamically adjusting the initial grasping path based on the deviation.

[0045] The ideal grasping position corresponds to a preset grasping feature of the target object; the preset grasping feature of the target object is determined based on an initial key feature point of the target object, or is a pre-stored grasping feature set by a user.

[0046] For example, the preset grasping features of the target object may be grasping features pre-stored and set by the user. For example, for a cubic object, the grasping features pre-stored and set by the user may be the edges and corners of the cube. For example, for a spherical object, the grasping features pre-stored and set by the user may be the equatorial position or the top of the spherical object.

[0047] Exemplarily, the grasping feature of the target object is determined based on the initial key feature points of the target object, that is, a grasping feature for guiding the robot arm to grasp is preset according to the initial key feature points extracted from the initial depth image of the target object. For example, if the target object is identified as a cube, its stable grasping point is automatically set according to its initial key feature points.

[0048] Then, in the embodiment of the present application, the deviation between the current position of the robot and the ideal grasping position is calculated based on the key feature points of the process and the preset grasping features of the target object; and then the initial grasping path is dynamically adjusted by the deviation. It should be noted that the deviation between the current position of the robot and the ideal grasping position is calculated. Then, the deviation is converted into motion control instructions for the robot joints through the control law, and the robot is driven to adjust the grasping.

[0049] That is, considering that in the actual grasping process, there may be factors involving the object itself that may lead to situations where high-precision grasping cannot be performed, such as deformation of easily deformed objects after the initial collection, or deformation during grasping that causes changes in the ideal grasping point position, thereby affecting the grasping success rate according to the initial grasping path. Therefore, in this application, the ideal grasping point position changes are analyzed in combination with the key feature points of the process, that is, based on the key feature points of the process and the preset grasping features of the target object, the deviation between the current position of the robot arm and the ideal grasping position is calculated; and then the initial grasping path is dynamically adjusted through the deviation. Through this real-time feedback mechanism, high-precision grasping of the robot arm can be achieved to further improve the success rate of grasping, that is, this method can reduce the risk of grasping failure or object damage caused by grasping position deviation.

[0050] In addition, in some dynamic environments, the position or shape of the target object may change. Therefore, by calculating the deviation in real time and adjusting the grasping path, the robotic arm can adapt to these changes and ensure the stability and success rate of grasping.

[0051] Optionally, the above-mentioned feature extraction network is set to include: a combination of a convolutional neural network sub-model and a recursive neural network; wherein the convolutional neural network sub-model is used to extract features of the depth image and obtain key feature points of the target object; the recursive neural network is used to receive key feature points of depth images of different time frames and track features of depth images of different time frames.

[0052] Alternatively, the feature extraction network is set to include: a combination of a convolutional neural network sub-model and a long short-term memory network; wherein the convolutional neural network sub-model is used to extract features of the depth image and obtain key feature points of the target object; the long short-term memory network is used to receive key feature points of depth images of different time frames and track features of depth images of different time frames.

[0053] That is, the embodiment of the present application provides a novel dynamic feature extraction network, which is composed of a convolutional neural network sub-model and a recursive neural network (RNN) or a long short-term memory network (LSTM).

[0054] The convolutional neural network sub-model adopts a multi-level architecture, including multiple convolutional layers, pooling layers, and fully connected layers. The design of the convolutional layer mainly enables it to extract features such as the edge and texture of the object.

[0055] The introduced recursive neural network or long short-term memory network structure is mainly used to process time series data, that is, it can remember the characteristic states of objects in different time frames, thereby realizing real-time tracking of the deformation state of the object.

[0056] For example, when an object undergoes deformation such as bending or twisting, the introduced recursive neural network or long short-term memory network structure can determine the degree and direction of the deformation based on previous feature information and current feature information.

[0057] Optionally, the feature extraction network is set to classify and mark newly appeared key feature points, and to relocate or remove disappeared or moved key feature points.

[0058] For example, the target object is a plastic rod (deformable and bendable). In the static state or during the grasping process, new key feature points will appear. Therefore, they need to be classified and marked, such as determining whether the key feature point belongs to the plastic rod. Of course, in the static state or during the grasping process, key feature points will disappear and move. At this time, it is necessary to combine recursive neural networks or long short-term memory network structures for tracking, and then achieve relocation or elimination.

[0059] Optionally, the method further includes: determining the degree of deformation of the target object based on key feature points corresponding to depth images of different frames; and controlling the grasping force of the robotic arm based on the degree of deformation of the target object.

[0060] It should be noted that this implementation can correspond to the robot arm being in an ideal grasping position, and during the grasping process, the deformation degree of the target object is analyzed to determine whether the target object can be grasped successfully. That is, this method indicates that when the robot arm is already in an ideal position, the grasping force can be adjusted according to the deformation degree of the target object to ensure that the grasping process will not damage the object due to excessive force, nor will it cause unstable grasping due to insufficient force.

[0061] Optionally, the trajectory model is set to be generated through training by a reinforcement learning algorithm; and the trajectory model is set to define a reward function.

[0062] The reward function is configured as follows: a positive reward is given when the movement of the robotic arm brings it closer to the sample object than the previous movement, or the grasping position does not cause a deformation degree greater than a set threshold to the sample object, or the grasping position does not cause damage to the sample object; a negative reward is given when the movement of the robotic arm causes it to move away from the sample object than the previous movement, or the grasping position causes a deformation degree greater than a set threshold to the sample object, or the grasping position causes damage to the sample object.

[0063] The training of the set trajectory model is explained below. In the embodiment of the present application, the grasping trajectory is optimized by reinforcement learning. First, the environment modeling and state definition can use the workspace of the manipulator and the state of the object as the environment of reinforcement learning. Define the position, speed, position, shape, deformation degree and other information of the manipulator joint as state variables, and determine the action space of the manipulator, including the rotation angle of the joint, the moving distance and other action instructions. Then, the reward function is designed. When the action of the manipulator makes it closer to the sample object than the previous action, or the grasping position does not cause a deformation degree greater than the set threshold (which can be set according to actual needs, and greater than the set threshold can be used as a judgment standard for excessive deformation) to the sample object, or the grasping position does not cause damage to the sample object, a positive reward is given; when the action of the manipulator makes it farther away from the sample object than the previous action, or the grasping position causes a deformation degree greater than the set threshold to the sample object, or the grasping position causes damage to the sample object, a negative reward is given. For example, the reward value is inversely proportional to the distance between the end effector of the manipulator and the center of the target object, and inversely proportional to the degree of deformation of the object. In the learning and optimization process, reinforcement learning algorithms (such as Q-learning, deep Q network, etc.) continuously learn and optimize the action strategy of the robot arm according to the environmental state and reward feedback. Through a large number of experiments and learning, the robot arm can select the optimal grasping trajectory under different initial states and object conditions, and complete the grasping of deformable objects in the fastest and most stable way.

[0064] Optionally, multiple tactile sensors are installed on the end effector of the robot arm; the method also includes: receiving detection pressure of the multiple tactile sensors; when the detection pressure of the multiple tactile sensors is greater than a first set pressure threshold, reducing the gripping force of the end effector of the robot arm; when the detection pressure of the multiple tactile sensors is less than a second set pressure threshold, increasing the gripping force of the end effector of the robot arm; wherein the second set pressure threshold is less than the first set pressure threshold.

[0065] The first set pressure threshold and the second set pressure threshold can be set according to needs, and there is no limitation on the specific values ​​of the thresholds. It should be noted that a plurality of tactile sensors are installed on the end effector of the robot arm, and these tactile sensors can sense the pressure, friction and other force information when the robot arm contacts the object. The distribution of the tactile sensors is designed according to the shape of the end effector and the requirements of the grasping task to ensure that the contact force information can be fully and accurately obtained.

[0066] It should be noted that when the detection pressure of multiple tactile sensors is greater than the first set pressure threshold, it indicates that the gripping force of the robot arm on the object is too large, which may cause damage to the object or unstable gripping. Therefore, the gripping force of the end effector of the robot arm can be reduced to reduce the pressure on the object.

[0067] When the detection pressure of multiple tactile sensors is less than the second set pressure threshold, it indicates that the gripping force of the robot arm on the object is too small or insufficient, which may cause the object to slip or fail to grasp. Therefore, the gripping force of the end effector of the robot arm can be increased to improve the stability of the gripping of the object.

[0068] Among them, changing the grasping force can be achieved through the motor, cylinder, etc. corresponding to the end effector of the robotic arm.

[0069] In addition, the above solution is mainly applicable to objects that are not easily deformed. For example, for metal objects, it is difficult to determine the grasping force directly through visual features. Therefore, it is necessary to combine tactile sensors to achieve accurate grasping of objects that are not easily deformed.

[0070] It can be seen that the fusion method of multiple tactile sensors provided in the embodiment of the present application can further improve the precise grasping and adjustment of objects that are not easily deformed, thereby making the present solution widely applicable to the grasping of easily deformable objects, irregular objects, and objects that are not easily deformed. For example, assuming that in an automated warehouse, there are many different types of objects, such as metal objects, cartons, plastic bottles, etc., the above methods can realize seamless switching grasping of different objects, and the grasping force can be changed through the multi-tactile sensor to achieve precise grasping.

[0071] Optionally, the method also includes: acquiring spectral information of the target object collected by the spectral sensor; determining the material of the target object based on the initial depth image and the spectral information; and determining the posture of the end effector of the robotic arm based on the material and shape of the target object.

[0072] That is, the embodiment of the present application provides an adaptive grasping posture planning.

[0073] Specifically, the material and shape of the object are identified by combining the spectral sensor. The material type of the object, such as metal, plastic, rubber, etc., is determined by analyzing the texture, color, reflectivity and other features of the object's surface. At the same time, the shape of the object is determined by the depth camera based on its geometric features such as contour and curvature.

[0074] Then, the grasping posture of the end effector of the robot arm can be adaptively planned according to the material and shape of the identified object. For objects of different materials, the contact angle and force distribution of the end effector are adjusted considering their hardness, elasticity and other characteristics. For example, for soft and deformable rubber objects, a larger contact area is used, and the grasping force is relatively uniform and gentle; for harder metal objects, a smaller area of ​​precise contact can be used, and the grasping force can be appropriately increased. According to the shape of the object, determine the optimal grasping position and direction of the end effector. For example, for spherical objects, choose a suitable wrapping grasping posture; for long objects, use a side clamping posture, etc., to ensure that the object can be stably held during the grasping process while avoiding unnecessary damage to the object.

[0075] Optionally, multiple tactile sensors are installed on the end effector of the robotic arm; the method also includes: determining a pressure threshold that matches the target object based on the material of the target object; receiving the detection pressure of multiple tactile sensors; and dynamically adjusting the gripping force of the end effector of the robotic arm based on the detection pressure of the multiple tactile sensors and the pressure threshold that matches the target object.

[0076] That is, the embodiment of the present application also provides a method for determining different pressure thresholds according to the materials of different target objects, thereby achieving targeted regulation and grasping.

[0077] Exemplarily, the target objects are cartons, plastic bottles, metals, etc., and the forces required are different due to the different materials. Therefore, in the embodiments of the present application, different materials of the target objects are adapted and different pressure thresholds are matched to provide more targeted regulation. That is, this method can adapt to goods of different materials, improving the versatility and flexibility of the robot arm. At the same time, since goods of different materials are adapted to different pressure thresholds, it is also safer during the grasping process.

[0078] See also Figure 3Based on the same inventive concept, the embodiment of the present application also provides a dynamic vision-based robotic arm grasping system 300, including: a first acquisition module 301, used to acquire an initial depth image acquired by a depth camera of a target object; a second acquisition module 302, used to determine the shape and position of the target object based on the initial depth image, and to extract features from the initial depth image based on a set feature extraction network to acquire initial key feature points of the target object; a path determination module 303, used to input the shape and position of the target object and the initial key feature points of the target object into a set trajectory model to determine the initial grasping path of the robotic arm; a grasping module 304, used to control the robotic arm to grasp the target object with the initial grasping path, and to dynamically adjust the initial grasping path to grasp during the grasping process; wherein, dynamically adjusting the initial grasping path includes: continuously acquiring a process depth image acquired by the depth camera of the target object, and to extract features from the process depth image based on the set feature extraction network to acquire process key feature points of the target object; and adjusting the initial grasping path based on the process key feature points.

[0079] See also Figure 4 Based on the same inventive concept, the present application embodiment provides a dynamic vision-based robotic arm grasping system 400, which is described from the hardware device and combined with the software level. The dynamic vision-based robotic arm grasping system 400 includes: a depth camera 401, a tactile sensor 402, a data processing and algorithm module 403, a control unit 404, and a robotic arm 405.

[0080] In terms of connection relationship, the depth camera 401 and the tactile sensor 402 are connected to the data processing and algorithm module 403 ; the data processing and algorithm module 403 is also connected to the control unit 404 ; and the control unit 404 is connected to the robotic arm 405 .

[0081] The depth camera 401 is responsible for collecting the image information of the object in real time and providing raw data for subsequent visual processing. The mechanical arm 405 is used as an actuator to complete the grasping action of the object. The tactile sensor 402 is used to sense the force information when the mechanical arm contacts the object to achieve force control. The control unit 404 coordinates the work between the modules to ensure the stable operation of the system. The data processing and algorithm module 403 is the core of the system, and the data processing and algorithm module 403 is used to implement the method steps in the aforementioned embodiment. It includes the improved visual servo algorithm, dynamic feature extraction network, reinforcement learning algorithm and adaptive grasping posture planning algorithm in the aforementioned embodiment. That is, real-time depth camera feedback and precise operation, through real-time depth camera feedback, the improved visual servo algorithm is used to achieve precise operation of deformable objects. It is more advanced than the traditional visual servo system and can handle more complex scenes and objects, especially in the grasping of irregular objects. The new dynamic feature extraction network, the designed new dynamic feature extraction network can track the deformation state of the object in real time, which is an improvement on the traditional feature extraction method, especially in the real-time tracking and analysis of the object state in a dynamic environment. Reinforcement learning optimizes the grasping trajectory. By optimizing the grasping trajectory through reinforcement learning, the system can learn and adapt autonomously to improve grasping efficiency and accuracy, which is more flexible and intelligent than traditional rule-based grasping strategies. Tactile feedback and force control. The system introduces tactile sensors to achieve force control of flexible objects through multi-sensor fusion to avoid deformation or damage during grasping. This is a significant improvement for traditional robotic arms. Adaptive grasping posture planning. The integrated adaptive grasping posture planning can dynamically adjust the posture and force of the end effector according to the material and shape of the object, greatly improving the flexibility and applicability of the system.

[0082] See also Figure 5 Based on the same inventive concept, the present application embodiment provides a module frame of an electronic device 500 that applies the above method. The electronic device 500 includes: at least one processor 501 ( Figure 5 Only one is shown in the figure), a memory 502, a computer program 503 stored in the memory 502 and executable on at least one processor 501, and when the processor 501 executes the computer program 503, the steps of the method in any of the foregoing embodiments are implemented.

[0083] The electronic device 500 can be a server, a personal computer, an industrial computer, etc.

[0084] Those skilled in the art will understand that Figure 5 The electronic device 500 is merely an example and does not constitute a limitation on the electronic device 500 , and may include more or less components than those shown in the figure, or may combine certain components, or may include different components.

[0085] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0086] In some embodiments, the memory 502 may be an internal storage unit of the electronic device 500, such as a hard disk or memory of the electronic device 500. In other embodiments, the memory 502 may also be an external storage device of the electronic device 500, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 500. Furthermore, the memory 502 may also include both an internal storage unit of the electronic device 500 and an external storage device.

[0087] It should be noted that the above-mentioned systems, devices, etc. are based on the same concept as the method embodiments of the present application. The modules designed for the systems, the steps performed by the devices, and the technical effects brought about can all be found in the method embodiment section and will not be repeated here.

[0088] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0089] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0090] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0091] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0092] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0093] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0094] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0095] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0096] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A robotic arm grasping method based on dynamic visual servoing, characterized in that: include: Obtain an initial depth image of the target object captured by a depth camera; Based on the initial depth image, determine the shape and position of the target object, and extract features from the initial depth image based on a set feature extraction network to obtain initial key feature points of the target object; Inputting the shape and position of the target object and the initial key feature points of the target object into a set trajectory model to determine the initial grasping path of the robot arm; Controlling the robot arm to grasp the target object along the initial grasping path, and dynamically adjusting the initial grasping path to grasp the target object during the grasping process; Among them, dynamically adjusting the initial grasping path includes: continuously acquiring the process depth image of the target object acquired by the depth camera, and extracting features from the process depth image based on the set feature extraction network to obtain the process key feature points of the target object; and adjusting the initial grasping path based on the process key feature points.

2. The robot arm grasping method based on dynamic visual servoing according to claim 1 is characterized in that: The adjusting the initial grasping path based on the key feature points of the process includes: Based on the key feature points of the process and the preset grasping features of the target object, calculating the deviation between the current position of the robot arm and the ideal grasping position; wherein the ideal grasping position corresponds to the preset grasping features of the target object; the preset grasping features of the target object are determined based on the initial key feature points of the target object, or are pre-stored grasping features set by the user; The initial grasping path is dynamically adjusted based on the deviation.

3. The robot arm grasping method based on dynamic visual servoing according to claim 1 is characterized in that: The set feature extraction network includes: a combination of a convolutional neural network sub-model and a recursive neural network; The convolutional neural network sub-model is used to extract the features of the depth image and obtain the key feature points of the target object; the recursive neural network is used to receive the key feature points of the depth images of different time frames and track the features of the depth images of different time frames; or, The set feature extraction network includes: a combination of a convolutional neural network sub-model and a long short-term memory network; Among them, the convolutional neural network sub-model is used to extract the features of the depth image and obtain the key feature points of the target object; the long short-term memory network is used to receive the key feature points of the depth images of different time frames and track the features of the depth images of different time frames.

4. The robot arm grasping method based on dynamic visual servoing according to claim 3 is characterized in that: The set feature extraction network is also used to classify and mark newly appearing key feature points, and to reposition or remove disappeared or moved key feature points.

5. The robot arm grasping method based on dynamic visual servoing according to claim 3 is characterized in that: The method further comprises: Determining the deformation degree of the target object based on key feature points corresponding to the depth images of different frames; Based on the deformation degree of the target object, the grasping force of the robotic arm is controlled.

6. The robot arm grasping method based on dynamic visual servoing according to claim 1 is characterized in that: The set trajectory model is generated by training through a reinforcement learning algorithm; The setting trajectory model defines a reward function; The reward function is configured as follows: a positive reward is given when the movement of the robotic arm brings it closer to the sample object than the previous movement, or the grasping position does not cause a deformation degree greater than a set threshold to the sample object, or the grasping position does not cause damage to the sample object; a negative reward is given when the movement of the robotic arm causes it to move away from the sample object than the previous movement, or the grasping position causes a deformation degree greater than the set threshold to the sample object, or the grasping position causes damage to the sample object.

7. The robot arm grasping method based on dynamic visual servoing according to claim 1 is characterized in that: A plurality of tactile sensors are installed on the end effector of the robotic arm; The method further comprises: receiving detection pressures of the plurality of tactile sensors; When the detection pressure of the plurality of tactile sensors is greater than a first set pressure threshold, reducing the gripping force of the end effector of the robotic arm; When the detection pressure of the plurality of tactile sensors is less than a second set pressure threshold, increasing the gripping force of the end effector of the robotic arm; The second set pressure threshold is smaller than the first set pressure threshold.

8. The robot arm grasping method based on dynamic visual servoing according to claim 1 is characterized in that: The method further comprises: Acquiring spectral information of the target object collected by a spectral sensor; Determine the material of the target object by combining the initial depth image and the spectral information; Based on the material of the target object and the shape of the target object, a posture of the end effector of the robot arm is determined.

9. The robot arm grasping method based on dynamic visual servoing according to claim 8 is characterized in that: A plurality of tactile sensors are installed on the end effector of the robotic arm; The method further comprises: Determining a pressure threshold matching the target object based on the material of the target object; receiving detection pressures of the plurality of tactile sensors; Based on the detection pressures of the multiple tactile sensors and a pressure threshold that matches the target object, the grasping force of the end effector of the robotic arm is dynamically adjusted.

10. A robot arm grasping system based on dynamic vision, characterized in that: include: A first acquisition module, used to acquire an initial depth image of a target object captured by a depth camera; A second acquisition module is used to determine the shape and position of the target object based on the initial depth image, and to extract features from the initial depth image based on a set feature extraction network to obtain initial key feature points of the target object; A path determination module, used to input the shape and position of the target object and the initial key feature points of the target object into a set trajectory model to determine the initial grasping path of the robot arm; A grasping module, used to control the robot arm to grasp the target object along the initial grasping path, and dynamically adjust the initial grasping path to grasp during the grasping process; Among them, dynamically adjusting the initial grasping path includes: continuously acquiring the process depth image of the target object acquired by the depth camera, and extracting features from the process depth image based on the set feature extraction network to obtain the process key feature points of the target object; and adjusting the initial grasping path based on the process key feature points.

Images