Object grabbing method, system and device, electronic equipment and storage medium
By deploying image acquisition equipment on the robotic arms and mechanical claws, collecting and analyzing images in real time, and controlling the rotation and elongation of the robotic arms, the problem of rapid and accurate control of the robotic arms and mechanical claws when grasping the target object is solved, and efficient object grasping effect is achieved.
Patent Information
- Application Number
- CN202311267646.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-27
- Publication Date
- 2025-03-28
AI Technical Summary
The prior art is difficult to achieve rapid and accurate control of the robotic arms and mechanical claws when grasping the target object, especially when the positional relationship of the target object is complex or the distance is uncertain.
By controlling the robotic arm to rotate at an initial length and using the image acquisition device deployed on the mechanical claw to acquire images in real time, it is determined whether the target object is within the center of the image. According to the image judgment, the robot arm is controlled to extend or stop rotation until the distance between the mechanical claw and the target object reaches a threshold value, and then the grab operation is performed.
It realizes rapid and accurate control of the robotic arm and mechanical claws when grabbing the target object, and can automatically adjust the length and rotation angle of the robotic arm when the target object position relationship is complex or the distance is uncertain, ensuring the accuracy and efficiency of the grasping.
Smart Images
Figure CN119704169A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of robotics, and in particular to an object grasping method, system, device, electronic device and storage medium. Background Art
[0002] In today's life, with the rapid development of science and technology, machines have gradually become popular in various industries to replace manpower to complete simple and repetitive work. The robotic arm is the most widely used automated mechanical device in the field of robotics. It can be seen in industrial manufacturing, medical treatment, entertainment services, military, semiconductor manufacturing, and space exploration. The robotic arm can simulate the movement of the human arm, and the mechanical claw connected to the robotic arm can grab the target object. Therefore, how to control the robotic arm and the mechanical claw to quickly and accurately grab the target object has become a key research direction. Summary of the invention
[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.
[0004] The first embodiment of the present disclosure provides an object grasping method, comprising:
[0005] Controlling the robotic arm to rotate at an initial length and acquiring an image acquired by an image acquisition device deployed on the robotic claw;
[0006] When the image contains a target object to be grasped and a first distance between a center point of the target object and a center point of the image is less than or equal to a first threshold, controlling the robot arm to stop rotating;
[0007] When the second distance between the mechanical claw and the target object is greater than a second threshold, the mechanical arm is controlled to extend, and the operation of controlling the mechanical arm to rotate and collect images is returned to be executed until the second distance between the mechanical claw and the target object is less than or equal to the second threshold;
[0008] Control the mechanical claw to grasp the target object.
[0009] Optionally, controlling the mechanical arm to rotate at an initial length comprises:
[0010] Controlling the robotic arm to rotate at the initial length and the first angle;
[0011] If the target object is not included in the image captured by the image acquisition device after one rotation, updating the first angle;
[0012] The mechanical arm is controlled to rotate at the initial length and the updated first angle until the target object is included in the image captured by the image acquisition device.
[0013] Optionally, the controlling the mechanical arm to rotate at an initial length further comprises:
[0014] When the target object is included in the image and a first distance between a center point of the target object and a center point of the image is greater than the first threshold, determining a positional relationship between a center point of the target object and a center point of the image;
[0015] Based on the positional relationship, the robotic arm is controlled to move in a direction approaching the target object with the initial length until the first distance in the image captured by the image acquisition device is less than or equal to the first threshold.
[0016] Optionally, also include:
[0017] The first threshold is determined according to the physical size of the target object.
[0018] Optionally, controlling the extension of the robotic arm includes:
[0019] When the second distance is greater than a third threshold, the robotic arm is controlled to extend based on the first adjustment amplitude; or,
[0020] When the second distance is less than or equal to a third threshold value and greater than the second threshold value, controlling the robot arm to extend based on a second adjustment amplitude;
[0021] The first adjustment range is greater than the second adjustment range.
[0022] Optionally, also include:
[0023] Obtaining an ultrasonic distance measuring device and an infrared distance measuring device deployed on the mechanical claw to respectively measure a third distance and a fourth distance between the mechanical claw and the target object;
[0024] When the third distance is less than or equal to the second threshold, or the fourth distance is less than or equal to the second threshold, it is determined that the second distance between the mechanical claw and the target object is less than or equal to the second threshold.
[0025] Optionally, after acquiring the image captured by the image acquisition device deployed in the middle of the mechanical claw, the method further includes:
[0026] Performing target recognition on the image to obtain candidate objects contained in the image and a first identifier corresponding to each candidate object;
[0027] In the case that any first identifier is the same as the second identifier of the target object, the candidate object corresponding to the any first identifier is determined to be the target object.
[0028] Optionally, before controlling the mechanical arm to rotate at an initial length, the method further includes:
[0029] Receive object grasping instructions;
[0030] The object grabbing instruction is parsed to obtain a second identifier corresponding to the target object to be grabbed.
[0031] The second aspect of the present disclosure provides an object grasping system, including: a controller, a mechanical arm, a mechanical claw, and an image acquisition device deployed on the mechanical claw, for executing the object grasping method described in the first aspect of the present disclosure and its possible implementation methods;
[0032] Wherein, the controller is used to control the movement of the robotic arm so that the robotic claw connected to the robotic arm approaches the target object;
[0033] The controller is also used to control the image acquisition device to acquire images during the movement of the robotic arm;
[0034] The controller is also used to control the mechanical claw to grasp the target object.
[0035] The third aspect of the present disclosure provides an object grasping device, comprising:
[0036] A control module, used for controlling the robot arm to rotate at an initial length;
[0037] An acquisition module, used to acquire images acquired by an image acquisition device deployed on the mechanical claw;
[0038] The control module is used to control the mechanical arm to stop rotating when the image contains a target object to be grasped and a first distance between a center point of the target object and a center point of the image is less than or equal to a first threshold;
[0039] The control module is further configured to control the mechanical arm to extend when the second distance between the mechanical claw and the target object is greater than a second threshold, and return to executing the operation of controlling the mechanical arm to rotate and collect images until the second distance between the mechanical claw and the target object is less than or equal to the second threshold;
[0040] The control module is also used to control the mechanical claw to grasp the target object.
[0041] In some embodiments, the control module is further configured to:
[0042] Controlling the robotic arm to rotate at an initial length and a first angle;
[0043] If the image captured by the image acquisition device does not contain the target object after one rotation, the first angle is updated;
[0044] The robotic arm is controlled to rotate at an initial length and an updated first angle until the image captured by the image acquisition device contains the target object.
[0045] In some embodiments, the control module is further configured to:
[0046] When the target object is included in the image and a first distance between a center point of the target object and a center point of the image is greater than a first threshold, determining a positional relationship between a center point of the target object and a center point of the image;
[0047] Based on the positional relationship, the robotic arm is controlled to move in a direction approaching the target object at an initial length until a first distance in an image captured by the image acquisition device is less than or equal to a first threshold.
[0048] In some embodiments, the system further includes a first determining module configured to:
[0049] A first threshold is determined according to the physical size of the target object.
[0050] In some embodiments, the control module is further configured to:
[0051] When the second distance is greater than the third threshold, the robot arm is controlled to extend based on the first adjustment amplitude; or,
[0052] When the second distance is less than or equal to the third threshold value and greater than the second threshold value, controlling the robot arm to extend based on the second adjustment amplitude;
[0053] The first adjustment range is greater than the second adjustment range.
[0054] In some embodiments, the system further includes a second determining module for:
[0055] Obtaining the third distance and the fourth distance between the mechanical claw and the target object measured by the ultrasonic ranging device and the infrared ranging device deployed on the mechanical claw respectively;
[0056] When the third distance is less than or equal to the second threshold, or the fourth distance is less than or equal to the second threshold, it is determined that the second distance between the mechanical claw and the target object is less than or equal to the second threshold.
[0057] In some embodiments, a third determining module is further included, which is used to:
[0058] Performing target recognition on the image to obtain candidate objects contained in the image and a first identifier corresponding to each candidate object;
[0059] In the case where any first identifier is identical to the second identifier of the target object, the candidate object corresponding to any first identifier is determined to be the target object.
[0060] In some embodiments, the system further includes an acquisition module for:
[0061] Receive object grasping instructions;
[0062] The object grabbing instruction is parsed to obtain a second identifier corresponding to the target object to be grabbed.
[0063] The fourth aspect embodiment of the present disclosure proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the object grasping method proposed in the first aspect embodiment of the present disclosure is implemented.
[0064] The fifth aspect embodiment of the present disclosure proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the object grasping method proposed in the first aspect embodiment of the present disclosure is implemented.
[0065] The object grasping method, system, device, electronic device and storage medium provided by the present disclosure have the following beneficial effects:
[0066] In the disclosed embodiment, the mechanical arm is controlled to rotate at an initial length, and an image captured by an image acquisition device deployed on the mechanical claw is acquired. After that, when the image contains the target object to be captured and the first distance between the center point of the target object and the center point of the image is less than or equal to the first threshold, the mechanical arm is controlled to stop rotating; when the second distance between the mechanical claw and the target object is greater than the second threshold, the mechanical arm is controlled to extend, and the operation of controlling the mechanical arm to rotate and capture images is returned to execute until the second distance between the mechanical claw and the target object is less than or equal to the second threshold, and finally the mechanical claw is controlled to capture the target object. Thus, the image acquisition device deployed on the mechanical claw can be used to capture the image of the mechanical arm during the rotation process in real time, and it can be determined whether the target object is within the image center range based on the image, and then the mechanical arm is extended. When the target object is in the image center and the second distance is less than the second threshold, the target object is captured, so that the target object can be captured quickly and accurately.
[0067] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0069] Figure 1 A schematic diagram of a flow chart of an object grasping method provided by an embodiment of the present disclosure;
[0070] Figure 2 A schematic flow chart of an object grasping method provided by another embodiment of the present disclosure;
[0071] Figure 2A A schematic diagram of a spherical coordinate system in which a robotic arm is located according to an embodiment of the present disclosure;
[0072] Figure 2B A schematic diagram of the positional relationship between a center point of a target object and a center point of an image provided by an embodiment of the present disclosure;
[0073] Figure 3 A schematic diagram of the structure of an object grasping system provided by an embodiment of the present disclosure;
[0074] Figure 4 A schematic diagram of the structure of an object grasping device provided by another embodiment of the present disclosure;
[0075] Figure 5 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0076] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.
[0077] The object grasping method, system, device, electronic device and storage medium of the embodiments of the present disclosure are described below with reference to the accompanying drawings.
[0078] Figure 1 A schematic flow chart of an object grasping method provided in an embodiment of the present disclosure.
[0079] The disclosed embodiment takes the object grasping method configured in an object grasping device as an example. The object grasping device can be applied to any electronic device so that the electronic device can perform an object grasping function.
[0080] like Figure 1 As shown, the object grasping method may include the following steps:
[0081] Step 101, controlling the robotic arm to rotate at an initial length, and acquiring an image captured by an image acquisition device deployed on the robotic claw.
[0082] Optionally, before controlling the robotic arm to rotate at the initial length, an object grabbing instruction may be received first, and then the object grabbing instruction may be parsed to obtain a second identifier corresponding to the target object to be grabbed.
[0083] The object grabbing instruction may be a text instruction input by the user, or a voice instruction of the user, etc., which is not limited in the present disclosure.
[0084] In some embodiments, voice input may be used first, and then speech-to-text may be used to extract the user's intent and form command characters to parse and grab the target object.
[0085] The second identifier may be a name, feature, etc. of the target object, which is not limited in the present disclosure.
[0086] Optionally, the initial length may be the length of the robot arm when receiving the object grabbing instruction, or may be a preset length of the robot arm when it is not in operation.
[0087] The image acquisition device may be a camera, a video camera or other device capable of acquiring images, and may acquire images at a certain frequency during the rotation of the robotic arm.
[0088] Among them, the image acquisition device can be deployed in the center of the mechanical claw, so that the image acquisition device can capture images directly in front of the mechanical claw.
[0089] Step 102 : When the image contains a target object to be grasped and a first distance between a center point of the target object and a center point of the image is less than or equal to a first threshold, control the robot arm to stop rotating.
[0090] Optionally, the first threshold may be a preset value, such as 1 cm, 2 cm, etc.
[0091] Optionally, the first threshold is determined according to the physical size of the target object. The larger the physical size, the larger the first threshold, but the first threshold cannot be greater than a preset value.
[0092] For distance, if the physical size of the target object is 5 cm, the first threshold can be 1 cm. If the physical size of the target object is 10 cm, the first threshold can be 1.5 cm. The first threshold cannot be greater than 2 cm.
[0093] Optionally, target recognition is performed on the image to obtain candidate objects contained in the image and a first identifier corresponding to each candidate object. Then, when any first identifier is the same as the second identifier of the target object, the candidate object corresponding to any first identifier is determined to be the target object, and the robotic arm is moved so that the first distance between the center point of the target object and the center point of the image is less than a first threshold.
[0094] Among them, a pre-trained target detection model can be used to perform target recognition on the image to obtain candidate objects contained in the image.
[0095] Among them, the target detection model will output the detection frame corresponding to the target object, so the center point of the detection frame can be determined as the center point of the target object.
[0096] In the embodiment of the present disclosure, after determining that the image contains the target object, the robotic arm can be controlled to move in a direction close to the target object, so that a first distance between the center point of the target object and the center point of the image is less than a first threshold.
[0097] Step 103, when the second distance between the robotic claw and the target object is greater than the second threshold, control the robotic arm to extend, and return to execute the operation of controlling the robotic arm to rotate and capture images until the second distance between the robotic claw and the target object is less than or equal to the second threshold.
[0098] In the disclosed embodiment, if the second distance between the mechanical claw and the target object is greater than the second threshold, it indicates that the mechanical claw is now far away from the target object, and if it grabs directly, it will not be able to grab. Therefore, it is necessary to extend the mechanical arm, control the rotation of the mechanical arm with the extended length, and make the first distance between the center point of the target object in the captured image and the center point of the image less than the first threshold, until the second distance between the mechanical claw and the target object is less than or equal to the second threshold.
[0099] Optionally, when the second distance is greater than the third threshold, the robot arm is controlled to extend based on the first adjustment amplitude. Alternatively, when the second distance is less than or equal to the third threshold and greater than the second threshold, the robot arm is controlled to extend based on the second adjustment amplitude.
[0100] The first adjustment range is greater than the second adjustment range. For example, the first adjustment range may be 5 cm, and the second adjustment range may be 3 cm.
[0101] It can be understood that the second distance is greater than the third threshold, which means that the distance between the mechanical claw and the object is relatively far. If the second adjustment range is used for adjustment, it needs to be adjusted many times, which is inefficient. Therefore, when the second distance is greater than the third threshold, the mechanical arm is controlled to extend based on the first adjustment range. When the second distance is less than or equal to the third threshold and greater than the second threshold, the mechanical arm is controlled to extend based on the second adjustment range. In this way, the mechanical arm can be controlled more reasonably and the efficiency of grasping the target object can be improved.
[0102] Optionally, an ultrasonic ranging device and an infrared ranging device deployed on the mechanical claw are obtained to respectively measure the third distance and the fourth distance between the mechanical claw and the target object, and then when the third distance is less than or equal to the second threshold, or the fourth distance is less than or equal to the second threshold, it is determined that the second distance between the mechanical claw and the target object is less than or equal to the second threshold.
[0103] It should be noted that the ultrasonic distance measuring device can measure a greater distance than the infrared distance measuring device, but the infrared distance measuring device is more accurate in measuring the distance of objects at close range. Therefore, the ultrasonic distance measuring device and the infrared distance measuring device can be deployed on the mechanical claw, and the second distance between the target object and the mechanical claw can be judged more accurately through the distance measurement results of the two distance measuring devices.
[0104] Step 104, controlling the mechanical claw to grasp the target object.
[0105] In the embodiment of the present disclosure, after determining that the second distance between the mechanical claw and the target object is less than or equal to the second threshold, the mechanical claw can be controlled to grasp the target object.
[0106] In the disclosed embodiment, the mechanical arm is controlled to rotate at an initial length, and an image captured by an image acquisition device deployed on the mechanical claw is acquired. After that, when the image contains the target object to be grasped and the first distance between the center point of the target object and the center point of the image is less than or equal to the first threshold, the mechanical arm is controlled to stop rotating; when the second distance between the mechanical claw and the target object is greater than the second threshold, the mechanical arm is controlled to extend, and the operation of controlling the mechanical arm to rotate and acquire images is returned to execute until the second distance between the mechanical claw and the target object is less than or equal to the second threshold, and finally the mechanical claw is controlled to grasp the target object. Thus, the image acquisition device deployed on the mechanical claw can be used to capture images of the mechanical arm in the process of rotation in real time, and it can be determined whether the target object is in the center range of the image based on the image, and then the mechanical arm is extended, and the target object is grasped when the target object is in the center of the image and the second distance is less than the second threshold, so that the target object can be grasped quickly and accurately.
[0107] Figure 2 A flowchart of an object grasping method provided by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the object grasping method may include the following steps:
[0108] Step 201, receiving an object grabbing instruction.
[0109] Step 202: Parse the object grabbing instruction to obtain a second identifier corresponding to the target object to be grabbed.
[0110] Step 203, controlling the robotic arm to rotate at an initial length and a first angle.
[0111] The first angle may be the angle between the robotic arm and the base when receiving an object grabbing instruction.
[0112] Figure 2A A schematic diagram of a spherical coordinate system in which a robotic arm is located according to an embodiment of the present disclosure. Figure 2A As shown in the figure, a spherical coordinate system is established based on the base of the robot arm as the origin. The straight-line distance from the robot claw to the base is the initial length (r). The robot claw is regarded as a point moving on a sphere with a radius of r. Its coordinates are (r, θ, φ), where θ is the angle between the radius r and the Z axis, that is, the difference between 90 degrees and the first angle, and φ is the angle between the projection r' of the radius r on the xy plane and the x axis. It can be seen that reducing or increasing θ can make the grasping head move north or south along the spherical surface, and reducing or increasing φ can make the grasping head move west or east along the spherical surface.
[0113] Step 204: if the image captured by the image capture device after one rotation does not contain the target object, the first angle is updated.
[0114] If the image captured by the image acquisition device after one rotation does not contain the target object, it means that the target object is not captured by the first angle movement, and the first angle needs to be changed. Specifically, the first angle can be increased or decreased by a preset angle adjustment range. This disclosure does not limit this.
[0115] Step 205 , controlling the robot arm to rotate at the initial length and the updated first angle until the image captured by the image acquisition device contains the target object.
[0116] It is understandable that the mechanical claw can move on a spherical surface with a radius as the initial length, so changing the first angle will definitely capture the target object. In the disclosed embodiment, the mechanical arm is controlled to rotate with a certain rotation logic, so that images containing the target object can be quickly and accurately captured.
[0117] Step 206 : when the image includes the target object and a first distance between the center point of the target object and the center point of the image is greater than a first threshold, determine the positional relationship between the center point of the target object and the center point of the image.
[0118] Figure 2B A schematic diagram of the positional relationship between the center point of a target object and the center point of an image provided by an embodiment of the present disclosure. Figure 2B As shown, the center point of the target object is (x1, y1), the center point of the image is the center of the field of view (x0, y0), and the center point of the target object is at the upper left of the center point of the image.
[0119] Step 207 , based on the positional relationship, control the robotic arm to move in a direction approaching the target object with an initial length until a first distance in the image captured by the image acquisition device is less than or equal to a first threshold.
[0120] In the embodiment of the present disclosure, after the positional relationship is determined, the robot arm can be controlled to move toward the target object with the initial length based on the positional relationship. For example, if the center point of the target object is at the upper left of the center point of the image, the robot arm stops rotating, and the robot arm is controlled to move with the initial length and the upper left.
[0121] Step 208, controlling the robotic arm to stop rotating.
[0122] Step 209, when the second distance between the robotic claw and the target object is greater than the second threshold, control the robotic arm to extend, and return to execute the operation of controlling the robotic arm to rotate and capture images until the second distance between the robotic claw and the target object is less than or equal to the second threshold.
[0123] Step 210, controlling the mechanical claw to grasp the target object.
[0124] The specific implementation forms of step 208 to step 209 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.
[0125] In the disclosed embodiment, the mechanical arm can be controlled to rotate with an initial length and a first angle; if the image captured by the image acquisition device does not contain the target object after one rotation, the first angle is updated; the mechanical arm is controlled to rotate with the initial length and the updated first angle until the image captured by the image acquisition device contains the target object, and when the image contains the target object and the first distance between the center point of the target object and the center point of the image is greater than a first threshold, the positional relationship between the center point of the target object and the center point of the image is determined, and based on the positional relationship, the mechanical arm is controlled to move in a direction close to the target object with the initial length until the first distance in the image captured by the image acquisition device is less than or equal to the first threshold. The mechanical arm is controlled to stop rotating; when the second distance between the mechanical claw and the target object is greater than the second threshold, the mechanical arm is controlled to extend, and the operation of controlling the mechanical arm to rotate and capture images is returned to be executed, until the second distance between the mechanical claw and the target object is less than or equal to the second threshold, and finally the mechanical claw is controlled to grasp the target object. Therefore, based on the length of the robotic arm and the first angle between the robotic arm and the base, the robotic arm can be regularly controlled so that the image acquisition device can quickly capture the target object, thereby further improving the efficiency of grasping the target object.
[0126] In some embodiments, users can use voice commands to let the robot complete object manipulation functions such as picking up and arranging desktop objects, or use voice-to-text conversion - text enters the cloud GPT large model for processing - output text-to-speech conversion to achieve continuous and natural dialogue between users and robots. The robot's form can be dynamically controlled and commanded to complete a series of specified actions.
[0127] In some embodiments, the robot can use the above-mentioned object grasping method to complete functions such as object grasping and desktop object arrangement.
[0128] In some embodiments, the robot can implement voice messages. Users can leave messages to the robot through voice, such as "I will go to the science park for a meeting and will be back around 11 o'clock." The robot will first convert the voice into text, and further analyze the intention behind the sentence, and store the corresponding conversation intention through the cloud GPT. When other people come to find the user, they can use voice inquiries to let the robot inform them of the user's current situation, thereby realizing the voice message function.
[0129] In some embodiments, the robot can realize intelligent dialogue. Through voice-to-text conversion, the text is uploaded to the cloud GPT large model, and the answer is generated, and the answer is converted into voice and played through the sound playback device, thereby realizing continuous and natural dialogue with the user.
[0130] In some embodiments, the robot can realize interactive entertainment. The main body of the intelligent networked robot is composed of a mechanical arm, which has at least 2 degrees of freedom, can realize rotation and bending, and the grasping head can perform grasping and opening and closing actions. By presetting fixed actions or action groups in advance and assisting with music, entertainment functions such as dancing and massage can be realized.
[0131] Figure 3 A schematic diagram of the structure of an object grasping system provided by an embodiment of the present disclosure; Figure 3 As shown, the object grasping system 300 includes: a controller 301, a mechanical arm 302, a mechanical claw 303, and an image acquisition device 304 deployed on the mechanical claw;
[0132] The controller is used to control the movement of the robotic arm so that the robotic claw connected to the robotic arm approaches the target object;
[0133] The controller is also used to control the image acquisition device to acquire images during the movement of the robotic arm;
[0134] The controller is also used to control the mechanical claw to grasp the target object.
[0135] In the disclosed embodiment, the object grasping system can execute the object grasping method, so that the image acquisition device deployed on the robotic claw can capture the image of the robotic arm during the rotation process in real time, and determine whether the target object is in the center of the image based on the image, and then extend the robotic arm to grasp the target object when the target object is in the center of the image and the second distance is less than the second threshold, thereby quickly and accurately grasping the target object.
[0136] In order to implement the above embodiments, the present disclosure also proposes an object grasping device.
[0137] Figure 4 A schematic diagram of the structure of an object grasping device provided in an embodiment of the present disclosure.
[0138] like Figure 4 As shown, the object grasping device 400 may include:
[0139] A control module 401 is used to control the robot arm to rotate at an initial length;
[0140] An acquisition module 402 is used to acquire an image acquired by an image acquisition device deployed on the mechanical claw;
[0141] A control module 401, configured to control the robot arm to stop rotating when the image contains a target object to be grasped and a first distance between a center point of the target object and a center point of the image is less than or equal to a first threshold;
[0142] The control module 401 is further configured to control the mechanical arm to extend when the second distance between the mechanical claw and the target object is greater than the second threshold, and return to executing the operation of controlling the mechanical arm to rotate and collect images until the second distance between the mechanical claw and the target object is less than or equal to the second threshold;
[0143] The control module 401 is also used to control the mechanical claw to grasp the target object.
[0144] In some embodiments, the control module 401 is further configured to:
[0145] Controlling the robotic arm to rotate at an initial length and a first angle;
[0146] If the image captured by the image acquisition device does not contain the target object after one rotation, the first angle is updated;
[0147] The robotic arm is controlled to rotate at an initial length and an updated first angle until the image captured by the image acquisition device contains the target object.
[0148] In some embodiments, the control module 401 is further configured to:
[0149] When the target object is included in the image and a first distance between a center point of the target object and a center point of the image is greater than a first threshold, determining a positional relationship between a center point of the target object and a center point of the image;
[0150] Based on the positional relationship, the robotic arm is controlled to move in a direction approaching the target object at an initial length until a first distance in an image captured by the image acquisition device is less than or equal to a first threshold.
[0151] In some embodiments, the system further includes a first determining module configured to:
[0152] A first threshold is determined according to the physical size of the target object.
[0153] In some embodiments, the control module 401 is further configured to:
[0154] When the second distance is greater than the third threshold, the robot arm is controlled to extend based on the first adjustment amplitude; or,
[0155] When the second distance is less than or equal to the third threshold value and greater than the second threshold value, controlling the robot arm to extend based on the second adjustment amplitude;
[0156] The first adjustment range is greater than the second adjustment range.
[0157] In some embodiments, the system further includes a second determining module for:
[0158] Obtaining the third distance and the fourth distance between the mechanical claw and the target object measured by the ultrasonic ranging device and the infrared ranging device deployed on the mechanical claw respectively;
[0159] When the third distance is less than or equal to the second threshold, or the fourth distance is less than or equal to the second threshold, it is determined that the second distance between the mechanical claw and the target object is less than or equal to the second threshold.
[0160] In some embodiments, a third determining module is further included, configured to:
[0161] Performing target recognition on the image to obtain candidate objects contained in the image and a first identifier corresponding to each candidate object;
[0162] In the case where any first identifier is identical to the second identifier of the target object, the candidate object corresponding to any first identifier is determined to be the target object.
[0163] In some embodiments, the system further includes an acquisition module for:
[0164] Receive object grasping instructions;
[0165] The object grabbing instruction is parsed to obtain a second identifier corresponding to the target object to be grabbed.
[0166] The functions and specific implementation principles of the above modules in the embodiments of the present disclosure can be referred to the above method embodiments, and will not be repeated here.
[0167] The object grasping device of the disclosed embodiment first controls the mechanical arm to rotate at an initial length, and obtains an image captured by an image acquisition device deployed on the mechanical claw, and then controls the mechanical arm to stop rotating when the image contains the target object to be grasped and the first distance between the center point of the target object and the center point of the image is less than or equal to the first threshold value; when the second distance between the mechanical claw and the target object is greater than the second threshold value, controls the mechanical arm to extend, and returns to execute the operation of controlling the mechanical arm to rotate and acquire images, until the second distance between the mechanical claw and the target object is less than or equal to the second threshold value, and finally controls the mechanical claw to grasp the target object. Thus, the image of the mechanical arm during the rotation process can be captured in real time by the image acquisition device deployed on the mechanical claw, and it can be judged whether the target object is in the center range of the image based on the image, and then the mechanical arm is extended, and the target object is grasped when the target object is in the center of the image and the second distance is less than the second threshold value, so that the target object can be grasped quickly and accurately.
[0168] In order to implement the above embodiments, the present disclosure further proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the object grasping method proposed in the above embodiments of the present disclosure is implemented.
[0169] In order to implement the above embodiments, the present disclosure further proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the object grasping method proposed in the above embodiments of the present disclosure is implemented.
[0170] Figure 5 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Figure 5 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0171] like Figure 5 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).
[0172] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.
[0173] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.
[0174] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the various embodiments of the present disclosure.
[0175] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described in the present disclosure.
[0176] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0177] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the methods mentioned in the above embodiments.
[0178] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0179] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0180] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.
[0181] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0182] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0183] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0184] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0185] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.
Claims
1. A method for grasping an object, characterized in that: include: Controlling the robotic arm to rotate at an initial length and acquiring an image acquired by an image acquisition device deployed on the robotic claw; When the image contains a target object to be grasped and a first distance between a center point of the target object and a center point of the image is less than or equal to a first threshold, controlling the robot arm to stop rotating; When the second distance between the mechanical claw and the target object is greater than a second threshold, the mechanical arm is controlled to extend, and the operation of controlling the mechanical arm to rotate and collect images is returned to be executed until the second distance between the mechanical claw and the target object is less than or equal to the second threshold; Control the mechanical claw to grasp the target object.
2. The method according to claim 1, characterized in that The controlling the mechanical arm to rotate at an initial length comprises: Controlling the robotic arm to rotate at the initial length and the first angle; If the target object is not included in the image captured by the image acquisition device after one rotation, updating the first angle; The mechanical arm is controlled to rotate at the initial length and the updated first angle until the target object is included in the image captured by the image acquisition device.
3. The method according to any one of claims 1 or 2, characterized in that: The controlling mechanical arm to rotate at an initial length also includes: When the target object is included in the image and a first distance between a center point of the target object and a center point of the image is greater than the first threshold, determining a positional relationship between a center point of the target object and a center point of the image; Based on the positional relationship, the robotic arm is controlled to move in a direction approaching the target object with the initial length until the first distance in the image captured by the image acquisition device is less than or equal to the first threshold.
4. The method according to claim 3, characterized in that Also includes: The first threshold is determined according to the physical size of the target object.
5. The method according to claim 1, characterized in that The controlling the extension of the robotic arm comprises: When the second distance is greater than a third threshold, the robotic arm is controlled to extend based on the first adjustment amplitude; or, When the second distance is less than or equal to a third threshold value and greater than the second threshold value, controlling the robot arm to extend based on a second adjustment amplitude; The first adjustment range is greater than the second adjustment range.
6. The method according to claim 1, characterized in that Also includes: Obtaining an ultrasonic distance measuring device and an infrared distance measuring device deployed on the mechanical claw, and measuring a third distance and a fourth distance between the mechanical claw and the target object respectively; When the third distance is less than or equal to the second threshold, or the fourth distance is less than or equal to the second threshold, it is determined that the second distance between the mechanical claw and the target object is less than or equal to the second threshold.
7. The method according to claim 1, characterized in that After acquiring the image acquired by the image acquisition device deployed in the middle of the mechanical claw, it also includes: Performing target recognition on the image to obtain candidate objects contained in the image and a first identifier corresponding to each candidate object; In the case that any first identifier is the same as the second identifier of the target object, the candidate object corresponding to the any first identifier is determined to be the target object.
8. The method according to claim 7, characterized in that Before controlling the mechanical arm to rotate at the initial length, the method further comprises: Receive object grasping instructions; The object grabbing instruction is parsed to obtain a second identifier corresponding to the target object to be grabbed.
9. An object grasping system, characterized in that: It comprises a controller, a mechanical arm, a mechanical claw and an image acquisition device deployed on the mechanical claw, and is used to perform the object grasping method according to any one of claims 1 to 8; Wherein, the controller is used to control the movement of the robotic arm so that the robotic claw connected to the robotic arm approaches the target object; The controller is also used to control the image acquisition device to acquire images during the movement of the robotic arm; The controller is also used to control the mechanical claw to grasp the target object.
10. An object grasping device, characterized in that: The device comprises: A control module, used for controlling the robot arm to rotate at an initial length; An acquisition module, used to acquire images acquired by an image acquisition device deployed on the mechanical claw; The control module is used to control the mechanical arm to stop rotating when the image contains a target object to be grasped and a first distance between a center point of the target object and a center point of the image is less than or equal to a first threshold; The control module is further configured to control the mechanical arm to extend when the second distance between the mechanical claw and the target object is greater than a second threshold, and return to executing the operation of controlling the mechanical arm to rotate and collect images until the second distance between the mechanical claw and the target object is less than or equal to the second threshold; The control module is also used to control the mechanical claw to grasp the target object.
11. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the object grasping method as claimed in any one of claims 1 to 8 is implemented.
12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the object grasping method as described in any one of claims 1 to 8 is implemented.