Vision-guided pick and place method, mobile robot, and computer-readable storage medium
Through visual guidance technology, the robot recognizes and matches object features in real time, determines the grasping position and vector, solving the problem of failed grabbing in the existing technology, achieving accurate grabbing and placement of target items, and improving the robot's adaptability and Accuracy.
Patent Information
- Application Number
- CN202211179290.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-27
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-09-27
AI Technical Summary
Existing robots find it difficult to cope with changes in shape, texture or position when grabbing objects, resulting in failed grabbing and unable to adapt to target items in random locations.
Using visual guidance method, the object image is taken in real time through the robotic arm and the camera, visual feature data is extracted to match the database, grasping position and vectors are determined, and the robotic arm is controlled to grab and place objects.
It achieves precise grasp and placement of target items, improves grasp accuracy, strong adaptability, and is suitable for a variety of application scenarios.
Smart Images

Figure CN115648176B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to robots, and more particularly to a vision-guided pick-and-place method, a mobile robot, and a computer-readable storage medium. Background Art
[0002] Grasp planning is a fundamental problem in the field of robotics that has attracted a growing number of researchers. Many robots perform "blind grasping" (e.g., in a warehouse or production line), where they aim to pick up an object from the same location every time. If there are any changes, such as the shape, texture, or position of the object, the robot will not know how to respond and the grasp attempt will most likely fail. This approach is obviously not suitable for other applications where the robot needs to grasp a target at a random location among other objects.
[0003] Therefore, it is necessary to provide a method and a mobile robot to overcome the above problems. Summary of the invention
[0004] Therefore, the present invention provides a method and a mobile robot to solve the above problems.
[0005] In order to achieve the above-mentioned objectives, the present invention provides a computer-implemented vision-guided picking and placing method for a mobile robot, the mobile robot comprising a robotic arm having a hand and a camera, the method comprising: receiving a command instruction commanding the mobile robot to grasp a target item in at least one object; controlling the mobile robot to move to a determined position associated with the at least one object, controlling the robotic arm to reach out to the at least one object, and using the camera to take one or more images of the at least one object in real time; extracting visual feature data from the one or more images, matching the extracted visual feature data with feature data of a target item preset in a database to identify the target item, and determining a grasping position and a grasping vector of the target item; and controlling the robotic arm and the hand to grasp the target item according to the grasping position and the grasping vector, and placing the target item at the target position.
[0006] Optionally, before receiving the command instruction, the method also includes: scanning at least one sample item associated with the pick-and-place task performed by the mobile robot to obtain at least one template image corresponding to each of the at least one sample item, wherein the at least one sample item includes the target item; inserting grasping data of at least one pair of grasping points into the at least one template image, wherein the grasping data includes a grasping position and a grasping vector associated with the at least one sample item; and storing the at least one template image and the grasping data corresponding to the at least one template image as feature data of a sample item corresponding to the at least one sample item.
[0007] Optionally, visual feature data is extracted from the one or more images, and the extracted visual feature data is matched with feature data of a target object preset in a database to identify the target object, including: using an oriented rapid rotation brief (ORB) algorithm to extract multiple visual feature data from each frame of the one or more images; and using a balanced binary search method and a nearest Euclidean distance algorithm to match the multiple visual feature data with visual feature data of the target object to identify the target object, wherein the visual feature data of the target object is extracted from a template image of the target object by the ORB algorithm.
[0008] Optionally, visual feature data is extracted from the one or more images, and the extracted visual feature data is matched with feature data of a target object preset in a database to identify the target object, including: using an oriented rapid rotation brief (ORB) algorithm to extract multiple visual feature data from each frame of the one or more images; comparing the multiple visual feature data with the visual feature data of the target object to obtain an inlier ratio and a perspective angle, wherein the visual feature data of the target object is extracted from a template image of the target object by the ORB algorithm; determining a confidence level of the match of the at least one object based on the inlier ratio and the perspective angle; and determining one of the at least one objects having the highest confidence score as the target object.
[0009] Optionally, determining the confidence level of the match of the at least one object based on the inlier ratio and the perspective angle includes: determining whether the inlier ratio is less than a first preset value; in response to the inlier ratio being not less than the first preset value, calculating the homography matrix of the one or more images, and applying the transformation matrix of the homography matrix to the shape of the target object in the template image; determining whether the area of the one or more images corresponding to the homography matrix exceeds a second preset value and is outside the field of view of the camera; in response to the area of the one or more images corresponding to the homography matrix not exceeding the second preset value and being outside the field of view of the camera, determining whether the shape distortion of the one or more images corresponding to the homography matrix is greater than a preset distortion; in response to the shape distortion of the one or more images corresponding to the homography matrix being not greater than a preset distortion, determining the confidence score of the match of the at least one object according to a preset confidence scoring rule, wherein the confidence scoring rule is positively correlated with the inlier ratio and the perspective angle.
[0010] Optionally, determining the grasping position and grasping vector of the target object includes: when the robotic arm reaches out to grab the target object, using the camera to capture multiple images of the target object; determining a first promising area in each of the multiple images of the target object based on the grasping position and grasping vector inserted in the at least one template image; controlling the robotic arm to move close to the first promising area, and determining a second promising area of the first promising area in each of the multiple images of the target object; and controlling the hand to move toward the target object according to the judgment result until the robotic arm grasps the target object.
[0011] Optionally, controlling the mobile robot to move to a determined position associated with the at least one object and controlling the robotic arm to reach for the at least one object includes: in response to detecting the target object, calculating a collision-free motion trajectory of the robotic arm based on the position and direction of the target object; in response to the existence of two or more collision-free motion trajectories, selecting one of the collision-free motion trajectories as a final motion trajectory; in response to the existence of a collision-free motion trajectory, selecting the collision-free motion trajectory as the final motion trajectory; and controlling the robotic arm to move toward the target object according to the final motion trajectory.
[0012] Optionally, the method also includes: using the camera to acquire a two-dimensional image of the target object, and determining depth information of the two-dimensional image based on the Euclidean distance between key feature positions in the two-dimensional image; and when the hand approaches the target object and stops above the target object, using a distance sensor to determine the distance information of the target object, and guiding the hand to grasp the target object based on the distance information and the depth information.
[0013] Optionally, before placing the target object at the target position, the method also includes: after the hand grasps the target object, controlling the hand to rotate the target object 360 degrees, and continuously observing the target object during the rotation process through a camera installed on one side of the hand; in response to detecting a barcode attached to the surface of the target object, continuously capturing an image of the barcode through the camera installed on the side of the hand; running a barcode search and decoding algorithm based on the image of the barcode to generate a barcode string; comparing the barcode string with a barcode string of a target product pre-stored in a database; and in response to the barcode string being the same as the pre-stored barcode string, controlling the mobile robot to place the target object at the target position.
[0014] The present invention also provides a mobile robot, comprising: a robotic arm, which includes a hand and a camera; one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include: instructions for receiving a command instruction commanding the mobile robot to grasp a target item in at least one object; instructions for controlling the mobile robot to move to a determined position associated with the at least one object, controlling the robotic arm to reach out for the at least one object, and using the camera to take one or more images of the at least one object in real time; instructions for extracting visual feature data from the one or more images, matching the extracted visual feature data with feature data of a target item preset in a database to identify the target item, and determining a grasping position and a grasping vector of the target item; and instructions for controlling the robotic arm and hand to grasp the target item according to the grasping position and the grasping vector, and placing the target item at the target position.
[0015] The present invention also provides a non-temporary computer-readable storage medium storing one or more programs to be executed by one or more processors of a mobile robot. When the one or more programs are executed by the one or more processors, the mobile robot performs processing including: receiving a command instruction commanding the mobile robot to grasp a target item in at least one object; controlling the mobile robot to move to a determined position associated with the at least one object, controlling the robotic arm to reach out for the at least one object, and using the camera to take one or more images of the at least one object in real time; extracting visual feature data from the one or more images, matching the extracted visual feature data with feature data of target items preset in a database to identify the target object, and determining a grasping position and a grasping vector of the target object; and controlling the robotic arm and hand to grasp the target object according to the grasping position and the grasping vector, and placing the target object at the target position.
[0016] The technical solution of the present invention has the following advantages: the mobile robot calculates the grasping position and grasping vector of the target object while approaching the target object, thereby realizing accurate calculation of the grasping action and improving the grasping accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments of the present disclosure and, together with the description, are further used to explain the principles of the present disclosure and enable those skilled in the relevant art to make and use the present disclosure. In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the specific embodiments or the description of the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.
[0018] Figure 1 is a perspective schematic diagram of a robotic arm according to an embodiment.
[0019] Figure 2 yes Figure 1 A top view of the robotic arm.
[0020] Figure 3 yes Figure 1 Side view of the robotic arm.
[0021] Figure 4 is a schematic diagram showing an exemplary application scenario of a mobile robot using a robotic arm.
[0022] Figure 5 is a schematic block diagram of a mobile robot according to one embodiment.
[0023] Figure 6 is an exemplary flow chart of a method for controlling a mobile robot to pick up and place objects according to one embodiment.
[0024] Figure 7 is a schematic diagram of a system for controlling a mobile robot according to one embodiment.
[0025] Figure 8 is an exemplary flow chart of a method for vision-guided pick and place of objects according to one embodiment.
[0026] Fig. 9 is an exemplary flow chart of a method for vision-guided pick and place of objects according to another embodiment.
[0027] Fig.10 is an exemplary flow chart of a method for determining one of the objects having a highest confidence score according to one embodiment.
[0028] Fig.11 is an exemplary flow chart of a method for determining one of the objects having a highest confidence score according to one embodiment.
[0029] Fig.12 An exemplary application scenario of a mobile robot's hand grasping a cylindrical target object is shown. DETAILED DESCRIPTION
[0030] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0031] The present disclosure is illustrated in the accompanying drawings by way of example and not limitation, in which like reference numerals represent like elements. It should be noted that reference to "one" embodiment in the present invention is not necessarily a reference to the same embodiment, and such reference may represent "at least one" embodiment.
[0032] Although specific configurations and arrangements are discussed, it should be understood that this is done for illustrative purposes only. Those skilled in the relevant art will recognize that other configurations and arrangements may be used without departing from the spirit and scope of the present disclosure. It is apparent to those skilled in the relevant art that the present disclosure may also be used in various other applications.
[0033] It should be noted that references in the specification to "one embodiment", "example embodiment", "some embodiments", "certain embodiments", etc. indicate that the described embodiments may include specific features, structures or characteristics, but each embodiment may not necessarily include the specific features, structures or characteristics. In addition, these phrases do not necessarily refer to the same embodiment. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it is within the knowledge of a technician in the relevant field to implement such feature, structure or characteristic in connection with other embodiments, whether or not explicitly described.
[0034] In general, terms can be understood, at least in part, from their use in context. For example, depending, at least in part, on the context, as used herein, the term "one or more" can be used to describe any feature, structure, or characteristic in a singular sense, or can be used to describe a combination of features, structures, or characteristics in a plural sense. Similarly, depending, at least in part, on the context, terms such as "one" or "the" can also be understood to convey singular usage or to convey plural usage. In addition, the term "based on" can be understood to not necessarily be intended to convey an exclusive set of factors, and can instead allow for the presence of additional factors that are not necessarily explicitly described, again depending, at least in part, on the context.
[0035] Although features and elements of the present disclosure are described as embodiments in particular combinations, each feature or element may be used alone or in various other combinations within the principles of the present disclosure, the use of which is to the greatest extent indicated by the broad, ordinary meaning of the terms expressing the appended claims.
[0036] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0037] Figure 1-3 A robotic arm 10 according to one embodiment is shown. The robotic arm 10 may be a foldable 6-DOF robotic arm. The robotic arm 10 may include a link body 11 including a plurality of links connected to each other, an end-of-arm tooling (EOAT) 12, and a mounting interface 13. The EOAT 12 may include a hand 121 and a camera 122 arranged on or near the hand 121. In one embodiment, the hand 121 may be a gripper with two fingers. However, the configuration of the hand 121 may be changed according to actual needs. For example, the hand 121 may be configured to be similar to a human hand. The robotic arm 10 may be connected to the body of a mobile robot via a mounting interface 13. In one embodiment, an embedded central processing unit (CPU) board of the mobile robot may execute an integrated visual manipulation algorithm to control the robotic arm 10 to perform a pick-and-place operation.
[0038] The mobile robot can be autonomous and versatile enough to perform precision grasps of small objects, as well as robust power grasps of heavier / larger objects. The robot is able to identify objects in the scene, train or detect the best grasping posture for each object geometry, locate the object relative to the robot base or EOAT 12, plan a collision-free or "crash-safe" path in the pre-grasp posture, plan a grasp configuration, and locate the object in the post-grasp configuration. With these capabilities, the mobile robot can provide unsupervised end-to-end logistics solutions to meet a variety of delivery and logistics needs. For example, Figure 4 As shown, after the mobile robot has moved to the target location (e.g., shelf 300), the robot arm 10 and the EOAT 12 are controlled to be in a state that enables the EOAT 12 to reach the determined location to pick up the determined object 400. In one embodiment, after the EOAT 12 grasps the object 400, the mobile robot will control the robot arm 10 and the EOAT 12 to place the object 400 to the determined location, such as a storage unit (e.g., a drawer) of the mobile robot. After the mobile robot moves from the shelf 300 to the determined location, the mobile robot will open the drawer 21 to allow the robot arm 10 to unload the object 400.
[0039] Figure 5 A schematic block diagram of a mobile robot 20 according to one embodiment is shown. The mobile robot 20 may be a wheeled robot that can work in various application environments, such as hospitals, factories, warehouses, shopping malls, streets, airports, homes, retirement centers, museums, restaurants, hotels, and even in the wild. However, the examples in the figure are only illustrative examples. The mobile robot 20 may be other types of robots. The mobile robot 20 may include an actuator 201, a mobile mechanism 202, a processor 203, a memory 204, and a communication interface module 205. The camera 122 is electrically connected to the processor 203 for transmitting the captured image to the processor 203. The mobile mechanism 202 may include one or more wheels and / or tracks. The actuator 201 is electrically coupled to the mobile mechanism 202 and the processor 203, and may drive the movement of the mobile mechanism 202 according to the instructions of the processor 203. The actuator 201 may be an electric motor or a steering gear. The memory 204 may include a non-temporary computer-readable storage medium. One or more executable computer programs 206 are stored in the memory 204. The processor 203 is electrically connected to the memory 204, and performs corresponding operations by executing an executable computer program stored in the memory 204. The communication interface module 205 may include a wireless transmitter, a wireless receiver, and a computer program executable by the processor 203. The communication interface module 205 is electrically connected to the processor 203 and is used for communication between the processor 203 and an external device. In one embodiment, the camera 122, the actuator 201, the moving mechanism 202, the processor 203, the memory 204, and the communication interface module 205 can be interconnected via a bus.
[0040] The processor 203 may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate, a transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor, etc. When the processor 203 executes the computer program 206, the steps in the embodiment of the control method of the mobile robot 20, such as Figure 1 Steps S81 to S85 in Figure 8 Steps S91 to S98 in Fig. 9 Steps S941 to S944 in are executed.
[0041] The memory 204 may be an internal storage unit of the mobile robot 20, such as a hard disk or a memory. The memory 204 may also be an external storage device of the mobile robot 20, such as a plug-in hard disk, a smart memory card (SMC) and a secure digital (SD) card, or any suitable flash memory card. In addition, the memory 204 may also include both an internal storage unit and an external storage device. The memory 204 is used to store computer programs, other programs, and data required by the mobile robot 20. The memory 204 may also be used to temporarily store data that has been output or is about to be output.
[0042] Exemplarily, the one or more computer programs 206 may be divided into one or more modules / units, and the one or more modules / units are stored in the memory 204 and can be executed by the processor 203. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the one or more computer programs 206 in the mobile robot 20.
[0043] Figure 6It is an exemplary flow chart of a method for controlling a mobile robot to pick up and place an object by visually guiding a manipulator 10. Specifically, first, a database is established in advance. Specifically, the sample of items to be picked up and placed by the mobile robot is determined, and then scanned to obtain template images taken from multiple different perspectives around each item sample. Each item sample corresponds to multiple template images. A grasping point is inserted into each template image. Here, the grasping point is a position on each item sample that can be contacted by the finger of the hand 121 so that the item sample can be grasped by the hand 121. The grasping data including the position associated with the grasping point is determined as metadata. The metadata and the corresponding template image are stored to obtain a database. The database can be configured in a cloud server or in a mobile robot 20. The mobile robot 20 then uses a camera 122 to capture an image. The captured image and the data in the database (including the metadata and template image of the item sample corresponding to the captured image) are input into the visual detection module for recognition. According to the recognition result, the mobile robot 20 controls the hand 121 to grasp the identified target item. After hand 121 grasps the identified target object, hand 121 rotates the object 360 degrees in the air. An ordinary camera (not a camera for detecting targets, for example, a universal serial bus (USB) camera) is installed on the side to observe the identified target object during the rotation. If the identified target object has a barcode on its surface, this USB camera continuously captures images of the identified target object and runs a barcode search and decoding algorithm to generate a barcode string. The result is compared with the database of the target to verify whether it is the correct object. After the barcode verification is passed, the mobile robot is controlled to place the target object at the target position, which is a pre-specified position corresponding to the pick and place task performed by the mobile robot. It should be noted that not every target object has a barcode. Only some ambiguous or error-prone ones have barcodes. The barcode stored in the database is entered in the object scanning step, and the template shooting and grasping data insertion are performed at the same time.
[0044] Figure 7It is a schematic diagram of a system for controlling a mobile robot. Specifically, a user can interact with a server through a mobile phone / tablet to control the operation of the mobile robot. The mobile phone / tablet and the server can exchange data through the http transmission protocol. The server can be connected to the arm-end computer through a router Ethernet with a master-slave relationship. The arm-end computer can be a C++ state machine and can include a visual real-time detection module and an operation control module. The arm-end computer can be connected to the camera of the mobile robot through a USB connector to obtain images taken by the camera. After the visual real-time detection module processes the image, it can determine the target object to be grasped. The operation control module then controls the hand of the mobile robot to grasp the target object. The arm-end computer can also communicate with a personal computer (PC) with a graphical user interface to realize the function of using a remote GUI program on the PC.
[0045] Figure 8 An exemplary flow chart of a method for vision-guided pick and place of objects according to one embodiment is shown. The method may be implemented to control Figure 4 and 5 The mobile robot 20 shown in FIG. 2 is equipped with a robotic arm having a hand and a camera to perform a pick-and-place operation. Specifically, the method can be performed by Figure 5 The method may be implemented by the mobile robot 20 shown in the figure or other control devices electrically coupled to the mobile robot 20. Other control devices may be, but are not limited to: desktop computers, tablet computers, laptop computers, multimedia players, servers, smart mobile devices (such as smart phones, handheld phones, etc.) and smart wearable devices (such as smart watches, smart glasses, smart cameras, smart bracelets, etc.) and other computer devices with computing and control functions. In one embodiment, the method may include steps S81 to S85.
[0046] Step S81: receiving a command instruction to instruct the mobile robot to grasp a target item in at least one object.
[0047] The mobile robot 20 can receive commands from a user. The user can send commands using a device (e.g., a mobile phone) wirelessly connected to the mobile robot 20. The command may include the location of at least one object (e.g., Figure 4 The mobile robot 20 can autonomously move to the position. The number of the at least one object is not limited, and in this embodiment, multiple objects are taken as an example. The target object is one of the multiple objects.
[0048] Step S82: Control the mobile robot to move to a determined position associated with the at least one object, control the robot arm to reach out to the at least one object, and use the camera to capture one or more images of the at least one object in real time.
[0049] After the mobile robot has moved to a certain position (e.g. Figure 4 After that, the mechanical arm 10 of the mobile robot 20 is controlled to reach out to reach multiple objects. When the mechanical arm 10 reaches the at least one object, the camera 122 is controlled to capture the image of the object in real time.
[0050] Step S83: extracting visual feature data from the one or more images, and matching the extracted visual feature data with preset feature data of the target object to identify the target object.
[0051] The robot performs visual correlation processing on the images captured by the camera 122. The robot needs at least distance information and light and dark information to determine the position and shape of the object. The visual features in each image may include: color features, texture features, shape features, spatial relationship features, etc. The extraction positions of the visual features may include edges, corners, regions and ridges.
[0052] The feature data of the target object may include an image of the target object in a pre-established database. A feature extraction algorithm may be executed to extract a plurality of visual feature data from the image of the target object. The extracted visual feature data will be compared with the visual feature data of a plurality of objects. Alternatively, the visual feature data of the target object may include not only the image of the target object pre-stored in the database, but also the visual feature data obtained after pre-processing the image of the target object, so that the visual feature data of the target object can be compared with the visual feature data of a plurality of objects.
[0053] The visual feature data of the extracted images of the multiple objects are compared with the feature data of the target objects in the database to identify the target object among the multiple objects. The object with the highest matching degree is determined as the target object.
[0054] Step S84: Determine the grasping position and grasping vector of the target object.
[0055] The gripping position is related to the type of the hand 121 of the mobile robot 20. The selection of the gripping position should take into account the convenience and firmness of the hand gripping the object. The gripping position is usually higher than the center of gravity of the object. The gripping vector is the gripping posture (i.e., the gripping direction). In one embodiment, the gripping vector is the yaw angle of the target object.
[0056] In one embodiment, the grasping position and grasping vector of the target object can be stored in the database as part of the characteristic data of the target object. The grasping vector of the target object can be calculated according to the current position relationship between the hand 121 and the target object to obtain the grasping angle at which the target object can be grasped and picked up.
[0057] Step S85: Control the robot arm and hand to grasp the target object according to the grasping position and the grasping vector, and place the target object at the target position.
[0058] The target location may be a location specified by the command instruction. Figure 4 After that, the mobile robot controls the robotic arm and hand to place the target item to the target location (e.g., the location on the shelf).
[0059] Fig. 9 A flow chart of a method for vision-guided picking and placing objects according to one embodiment is shown. The method may include steps S91 to S96.
[0060] Step S91: Establish a database of objects to be picked up and placed by the mobile robot.
[0061] The database can be established on the server. The scanning device is used to scan the object samples related to the pick-and-place tasks performed by the mobile robot. Thus, the template image of the object sample is obtained. The object sample is the object that the mobile robot may grab when performing the pick-and-place task. The template image is an image obtained by scanning the object sample from different angles by the scanning device. Each object sample corresponds to a plurality of template images captured at different viewing angles, so that the target object can be identified by comparing one or more images of the target object captured at different viewing angles with the corresponding template image.
[0062] The gripping points of the multiple template images of each item sample are then determined using a preset algorithm. The gripping data of these gripping points are inserted into each template image. For example, the gripping data of at least one pair of gripping points are inserted into at least one template image. The gripping data may include the position and gripping vector of the hand 121 that grasps the corresponding item sample. The mechanical arm of the mobile robot may firmly grasp the object according to the gripping data corresponding to the object. The gripping data is used as metadata. The multiple template images of each item sample and the metadata corresponding to each template image are stored as feature data of the corresponding item sample to establish a database. The item sample includes a target item to be grasped.
[0063] To complete the end-to-end object recognition and detection tasks, an item scanning system is included as an additional system, where an automated method collects the desired training data about the target items. This item scanning system is used to scan the items in the deployment site, so the pick and place system is customized for specific application needs, as different logistics applications need to handle different target items.
[0064] Step S92: Receive command instructions.
[0065] The command instruction is used to instruct the mobile robot 20 to grab a target item from a plurality of objects. The mobile robot 20 may receive a command instruction from a user. The user may send a command instruction using a device (e.g., a mobile phone) wirelessly connected to the mobile robot 20. The command instruction may include the location of the plurality of objects (e.g., Figure 4 shelf), so that the mobile robot 20 can move to this position autonomously.
[0066] Step S93: Control the mobile robot to move and capture images of the object according to the command instructions.
[0067] After receiving the command instruction, the mobile robot autonomously moves to a determined position associated with the target object. Then, the mobile robot controls the mechanical arm to reach the object and uses the camera to capture the image of the object in real time.
[0068] In one embodiment, after the mobile robot identifies the target object, it calculates the collision-free motion trajectory of the manipulator according to the position and orientation of the target object. The mobile robot controls the motion of the manipulator according to the calculation results. Specifically, the motion planner is executed to check whether the inverse kinematics (IK) solution can be calculated, and whether any predefined self-collision can be avoided in the trajectory solution calculated based on the IK solution. If there are multiple solutions, one of the calculated motion trajectories will be randomly selected as the final motion trajectory. If there is only one solution, the calculated motion trajectory is selected as the final motion trajectory. The manipulator will be controlled to grab the target object according to the final trajectory.
[0069] If there is more than one solution, the final trajectory will be executed by the low-level robot motion control. The robot will move to the object detection and recognition site, open the EOAT, wait for detailed position / orientation commands calculated by the vision module, and approach the object and grasp it with the EOAT half-open.
[0070] Low-level robotic arm motion control refers to position control and velocity control, while high-level motion control can include task scheduling and interaction with other modules such as computer vision and voice commands.
[0071] In one embodiment, the mobile robot may include an active lighting control module mounted on the robot body. The active lighting control module is to ensure that the visual detection performance does not decrease in non-optimal lighting environments such as darkness and uneven lighting.
[0072] Step S94: extracting visual feature data from the image, and matching the extracted visual feature data with the feature data of the target object to identify the target object.
[0073] For each frame of the object image captured by the camera, the directional rapid rotation brief (ORB) algorithm can be used to extract data of multiple visual features. In one embodiment, up to 200 visual feature data can be extracted. Then, the multiple visual feature data are matched with the visual feature data of the target object using a balanced binary search method and a nearest Euclidean distance algorithm to identify the target object.
[0074] In one embodiment, the ORB algorithm can be used to extract the visual feature data of the target object from the template image of the target object stored in the above-mentioned database. Among all objects, the object with the visual feature with the smallest Euclidean distance to the visual feature of the target object in the database is the matching target object to be grasped. Then the position information and posture estimation (i.e., direction) of the target object are calculated. In order to speed up the matching calculation, the balanced binary search method "KD tree" is used to speed up the search process. For multi-core CPU platforms, feature extraction and feature matching are optimized through multi-threaded parallelization. Depending on the clutter of the background, each frame of the image requires 50-100 milliseconds of calculation time.
[0075] Reference Fig.10 In one embodiment, step S94 may include the following steps.
[0076] Step S941: extract multiple visual feature data from each frame of image using the Oriented Rotation Brief (ORB) algorithm.
[0077] Step S942: Compare the visual feature data with the visual feature data of the target object to obtain the inlier ratio and the perspective angle. The visual feature data of the target object is extracted from the template image of the target object using the ORB algorithm.
[0078] Step S943: Determine the confidence level of the object matching degree based on the inlier ratio and the perspective angle.
[0079] Step S944: Determine one of the objects with the highest confidence score as the target item.
[0080] False positives can be filtered out by scoring the confidence of the match. The detected objects are sorted and the object with the highest confidence score is determined as the target object. Generally, there are two methods to score the confidence.
[0081] Specifically, the first method is to determine the confidence score based on the inlier ratio. That is, the feature points of each object in the captured image are compared with the feature points of the target object stored in the database. The higher the inlier ratio, the higher the confidence score.
[0082] The second method is to determine the confidence score based on the perspective angle. That is, the distortion of the objects in the captured images relative to the target items stored in the database is compared with each other. The smaller the perspective angle, the higher the confidence score.
[0083] Fig.11 An exemplary flow chart of a method for determining one of the objects with the highest confidence score as a target item is shown.
[0084] Specifically, the processor 203 determines whether the inlier ratio is less than a first preset value. When the inlier ratio is not less than the first preset value, the processor 203 calculates the homography matrix of the image captured by the camera 122, and applies the transformation matrix of the homography matrix to the shape of the target object in the template image. The processor 203 determines whether the image corresponding to the homography matrix has an area exceeding the second preset value outside the field of view of the camera. When the area of the image corresponding to the homography matrix does not exceed the second preset value and is not within the field of view of the camera, the processor 203 determines whether the shape distortion of the image corresponding to the homography matrix exceeds the preset distortion. When the shape distortion of the image corresponding to the homography matrix is not greater than the preset distortion, the processor 203 determines the confidence score of the object matching degree according to the preset confidence scoring rule. The confidence scoring rule is positively correlated with the inlier ratio and the perspective angle.
[0085] In other words, the processor 203 compares the captured image with the image of the target object to be grasped in the database to determine whether there are too few inlier matches. If not, the homography matrix of the captured image is calculated, and the transformation matrix of the homography matrix is applied to the shape of the object to be grasped. According to the homography matrix and the transformation matrix, it can be determined whether most of the area of the image corresponding to the homography matrix is outside the field of view of the camera. If not, the processor 203 determines whether the shape of the image corresponding to the homography matrix is severely distorted. If not, the processor 203 calculates a confidence score, which is mainly determined by the inlier ratio and the perspective angle. The respective weights can be set according to actual needs. In an example, the calculation can be performed by the following C language code: float regionScore = 1-abs(90-minAngle) / 90.0; float inlinerPerc = getInliers.size / (getInliers.size+getOutliers.size); float confScore = 1*regionScore*inlinerPerc / (regionScore+inlinerPerc), where "float" represents a decimal, "regionScore" represents the degree of distortion of the detection area, and each template image is a rectangular image. If the homography is still a rectangle and each corner is equal to 90 degrees, it means there is no distortion. In this case, regionScore is equal to 100%. If the distortion is large, such as becoming a trapezoid, the smallest of the four corners is equal to 45 degrees, then regionScore = 1–(90-45) / 90 = 50%. "minAngle" represents the minimum inner angle of the distorted image, and "inlinerPerc" represents the ratio of the inner point to all detection points. For example, each template image has 100 points and the homography has 40 corresponding points. Therefore, inlier is equal to 40, outlier is equal to 60, and inlinerPerc = 40%. "confScore" represents the confidence score, which is determined by regionScore and inlinerPerc. The larger the regionScore and inlinerPerc, the larger the confScore.
[0086] Step S95: Determine the grasping position and grasping vector of the target object.
[0087] When the mobile robot is ready to grasp the target object, it performs a visual manipulation closed-loop optimization. When the hand with the camera is close to the target object, multiple inspections are performed through the vision-guided algorithm, and multiple images are inspected on the timeline. The mobile robot's hand is controlled to move to the "promising area" through vision guidance, and further inspections are performed in the promising area. Each inspection finds the best grasping position and grasping vector for the promising area, and then controls the mobile robot's hand to approach the target object based on the inspection results. After that, another inspection is performed, and then the mobile robot's hand is controlled to approach the target object further, until the hand successfully grasps the target object based on the latest grasping position and grasping vector, completing the visual manipulation closed-loop optimization.
[0088] In other words, when the robot arm reaches out to the target object, the camera captures multiple images of the target object. Based on the grasping position and grasping vector inserted into the template image, the first promising area of each image of the target object is determined. After that, the robot arm is controlled to approach the first promising area, and the second promising area of the first promising area of each image of the target object is determined. Then, based on the judgment result, the robot hand is controlled to move toward the target object until the robot arm grasps the target object. The judgment result here refers to the grasping position and grasping vector detected when the distance between the hand and the target object is less than the preset distance.
[0089] Step S96: Control the robot arm and hand to grasp the target object according to the grasping position and the grasping vector, and place the target object at the target position.
[0090] In one embodiment, a distance sensor can be mounted on the hand of a mobile robot. The camera of the mobile robot is used to capture a two-dimensional image, and the depth information of the two-dimensional image is estimated by the Euclidean distance between the positions of key features in the two-dimensional image. The shorter the distance means the smaller the observation value, and therefore the farther the target is relative to the camera. The accuracy of the depth estimation result is within 0.5cm. When the hand approaches and stops above the object, the distance sensor will help guide the hand of the mobile robot to the precise depth level where the surface of the object is located. The distance sensor detects the distance from the mobile robot hand to the target object, and guides the hand to the surface of the target object based on the distance. The camera and the distance sensor set on the hand jointly determine the grasping depth of the hand grasping the target object.
[0091] Fig.12 2 shows exemplary detection results of a target object obtained during vision-guided detection. Specifically, Fig.12 The dotted line in represents the grasping contour line around the exemplary target object (ie, cylindrical object). The horizontal grasping axis of the target object is to accommodate the grasping balance of the two-fingered hand. The intersection of the grasping axis and the grasping contour line is the grasping position of the two-fingered hand.
[0092] Step S97: Perform barcode verification to verify whether the desired target item has been grasped.
[0093] In one embodiment, a barcode verification module is integrated on a mobile robot to ensure nearly 100% accuracy and minimal misidentification rate. Specifically, after the hand holds the target object, the hand is controlled to rotate the target object 360 degrees in the air, and the barcode verification module continuously observes the target object during the rotation process through a USB camera installed on one side of the hand. If a barcode is attached to the surface of the target object, the barcode verification module continuously collects barcode images through the USB camera, runs a barcode search and decoding algorithm based on the collected barcode image, and generates a barcode string. The barcode verification module compares the generated barcode string with the barcode string of the target object pre-stored in the database. If the same, it is determined that the object grasped by the mobile robot is the target grasping object of this grasping task. Then the mobile robot is controlled to place the target object to a determined target position. In one embodiment, only some ambiguous or error-prone objects have barcodes. If there is no barcode on the target object, the mobile robot can be controlled to directly place the target object to a determined target position without barcode verification. In the object scanning step, the barcode stored in the database is input, and the template is photographed and the grasping data is inserted at the same time. Further, if the generated barcode string is different from the barcode string of the pre-stored target object, it is determined that the object grasped by the mobile robot is not the expected grasping target object of this grasping task. In this case, the hand can be controlled to place the grasped object in another preset position or put the grasped object back to its original place.
[0094] Step S98: In response to the verification being passed, the target object is placed at the target location.
[0095] After the barcode verification is passed, the mobile robot is controlled to move to the determined position, release its grip, and place the grabbed target object at the target position.
[0096] By implementing the method of the above embodiment, the mobile robot uses its camera to capture images of multiple objects, extracts visual features from the images, matches the extracted visual features with the data of the target object in the database, and determines the target object to be grasped. The mobile robot uses the mobility of the camera installed on the EOAT to perform a closed-loop optimization of "looking and approaching". The mobile robot adopts a multiple detection method. That is, the mobile device calculates the grasping position and grasping vector of the target object while approaching the target object, thereby realizing the accurate calculation of the grasping action and improving the grasping accuracy. When the target object has been grasped, the barcode is verified to further improve the accuracy of grasping the target object.
[0097] By implementing the method of the above embodiment, the mobile robot can provide an end-to-end complete solution for the automation of item distribution. The system is complete, including from item scanning to delivery verification. In addition, the visual and operating software can be run on a single central processing unit (CPU) board. The automation of the delivery process helps to improve the above situation. Technically, the mobile robot is self-contained, including an item scanning device, barcode verification, and internal lighting design, with minimal modification work on the target logistics site, which is cost-effective.
[0098] In one embodiment, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium can be configured in the mobile robot 20. The non-transitory computer-readable storage medium stores an executable computer program, and when the program is executed by one or more processors of the mobile robot 20, the method of visually guided picking and placing objects described in the above embodiments is implemented.
[0099] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0100] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0101] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0102] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0104] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0105] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0106] It is intended that the specification and examples be considered as exemplary only, with a true scope being indicated by the following claims and their equivalents.
[0107] For the purpose of explanation, the above description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. In view of the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best explain the principles of the disclosure and its practical application, thereby enabling others skilled in the art to best utilize the disclosure and various embodiments with various modifications suitable for the intended specific use.
Claims
1. A computer-implemented vision-guided pick and place method for a mobile robot including a robotic arm having a hand and a camera, the method comprising: receiving a command instruction for instructing the mobile robot to grasp a target item in at least one object; Controlling the mobile robot to move to a determined position associated with the at least one object, controlling the robot arm to reach for the at least one object, and using the camera to capture one or more images of the at least one object in real time; Extracting visual feature data from the one or more images, matching the extracted visual feature data with feature data of a target object preset in a database to identify the target object, and determining a grasping position and a grasping vector of the target object, wherein the grasping position is determined based on the type of the hand of the robotic arm and is higher than the center of gravity of the target object, and the grasping vector includes a grasping posture; and Control the robot arm and hand to grasp the target object according to the grasping position and the grasping vector, and place the target object at the target position; Before receiving the command instruction, the method further includes: Scanning at least one sample object associated with a pick-and-place task performed by the mobile robot to obtain a plurality of template images corresponding to each of the at least one sample object, wherein the at least one sample object includes the target object, the plurality of template images being taken from a plurality of different viewing angles around each sample object; inserting grasping data of at least one pair of grasping points into each template image, wherein the grasping data includes a grasping position and a grasping vector associated with the at least one sample item, the at least one pair of grasping points being located at locations on each sample item that contact fingers of the hand so that each sample item can be grasped by the hand; and storing the template image and the grasping data corresponding to the template image as feature data of a sample object corresponding to the at least one sample object; The step of extracting visual feature data from the one or more images and matching the extracted visual feature data with feature data of a target object preset in a database to identify the target object includes: extracting a plurality of visual feature data from each frame of the one or more images using a directional fast rotation brief algorithm; and Matching the plurality of visual feature data with visual feature data of the target object using a balanced binary search method and a nearest Euclidean distance algorithm to identify the target object, wherein the visual feature data of the target object is extracted from a template image of the target object by a directional fast rotation brief algorithm; Wherein, determining the grasping position and grasping vector of the target object includes: When the robot arm reaches out to grab the target object, the camera is used to capture multiple images of the target object; determining a first promising region of each of the plurality of images of the target object based on the grasping position and the grasping vector inserted into the template image; Control the robot arm to move close to the first promising area, and determine a second promising area of the first promising area of each of the multiple images of the target object by performing multiple detections in the first promising area, find the best grasping position and the best grasping vector for the first promising area in each detection, and control the hand to approach the target object according to the best grasping position and the best grasping vector after each detection; The hand is controlled to move toward the target object until the robot arm grasps the target object according to the judgment result, wherein the judgment result refers to the grasping position and grasping vector detected when the distance between the hand and the target object is less than a preset distance.
2. The method according to claim 1, characterized in that Controlling the mobile robot to move to a determined position associated with the at least one object and controlling the robot arm to reach for the at least one object comprises: In response to detecting the target object, calculating a collision-free motion trajectory of the robot arm based on the position and orientation of the target object; In response to the existence of two or more collision-free motion trajectories, one of the collision-free motion trajectories is selected as the final motion trajectory; In response to the existence of a collision-free motion trajectory, selecting the collision-free motion trajectory as the final motion trajectory; The robot arm is controlled to move toward the target object according to the final motion trajectory.
3. The method according to claim 2, characterized in that Also includes: Using the camera to acquire a two-dimensional image of the target object, and determining depth information of the two-dimensional image according to the Euclidean distance between key feature positions in the two-dimensional image; and When the hand approaches the target object and stops above the target object, the distance sensor is used to determine the distance information of the target object, and the hand is guided to grasp the target object according to the distance information and depth information.
4. The method according to claim 1, characterized in that: Before placing the target object at the target location, it also includes: After the hand grasps the target object, the hand is controlled to rotate the target object 360 degrees, and the target object in the rotating process is continuously observed through a camera installed on one side of the hand; In response to detecting a barcode attached to the surface of the target object, continuously capturing images of the barcode via the camera mounted on the side of the hand; running a barcode lookup and decoding algorithm based on the image of the barcode to generate a barcode string; Compare the barcode string with the barcode string of the target product pre-stored in the database; In response to the barcode string being identical to the pre-stored barcode string, the mobile robot is controlled to place the target object at the target position.
5. A mobile robot comprising: a robotic arm, which includes a hand and a camera; one or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising: receiving an instruction for commanding the mobile robot to grasp a target item in at least one object; Instructions for controlling the mobile robot to move to a determined position associated with the at least one object, controlling the robot arm to reach for the at least one object, and using the camera to capture one or more images of the at least one object in real time; Extracting visual feature data from the one or more images, matching the extracted visual feature data with feature data of a target object preset in a database to identify the target object, and determining a grasping position and a grasping vector of the target object; and An instruction for controlling the robot arm and hand to grasp the target object according to the grasping position and the grasping vector, and placing the target object at the target position; Before receiving the command instruction, the one or more programs further include: instructions for scanning at least one sample item associated with a pick-and-place task performed by the mobile robot to obtain a plurality of template images corresponding to each of the at least one sample item, wherein the at least one sample item includes the target item, the plurality of template images being taken from a plurality of different viewing angles around each sample item; instructions for inserting into each template image grip data for at least one pair of grip points, wherein the grip data comprises a grip position and a grip vector associated with the at least one sample item, the at least one pair of grip points being located at locations on each sample item that contact fingers of the hand so that each sample item can be gripped by the hand; and An instruction to store the template image and the grasping data corresponding to the template image as feature data of a sample object corresponding to the at least one sample object; The instruction to extract visual feature data from the one or more images and match the extracted visual feature data with feature data of a target object preset in a database to identify the target object includes: instructions for extracting a plurality of visual feature data from each frame of the one or more images using a directional fast rotation brief algorithm; and instructions for matching the plurality of visual feature data with visual feature data of the target object using a balanced binary search method and a nearest Euclidean distance algorithm to identify the target object, wherein the visual feature data of the target object is extracted from a template image of the target object by a directional fast rotation brief algorithm; Wherein, determining the grasping position and grasping vector of the target object includes: When the robot arm reaches out to grab the target object, the camera is used to capture multiple images of the target object; determining a first promising region of each of the plurality of images of the target object based on the grasping position and the grasping vector inserted into the template image; Control the robot arm to move close to the first promising area, and determine a second promising area of the first promising area of each of the multiple images of the target object by performing multiple detections in the first promising area, find the best grasping position and the best grasping vector for the first promising area in each detection, and control the hand to approach the target object according to the best grasping position and the best grasping vector after each detection; The hand is controlled to move toward the target object until the robot arm grasps the target object according to the judgment result, wherein the judgment result refers to the grasping position and grasping vector detected when the distance between the hand and the target object is less than a preset distance.
6. A non-transitory computer-readable storage medium storing one or more programs to be executed by one or more processors of a mobile robot, the one or more programs, when executed by the one or more processors, causing the mobile robot to perform a process comprising: receiving a command instruction for instructing the mobile robot to grasp a target item in at least one object; Controlling the mobile robot to move to a determined position associated with the at least one object, controlling the robot arm to reach for the at least one object, and using the camera to capture one or more images of the at least one object in real time; Extracting visual feature data from the one or more images, matching the extracted visual feature data with feature data of a target object preset in a database to identify the target object, and determining a grasping position and a grasping vector of the target object, wherein the grasping position is determined based on the type of the hand of the robotic arm and is higher than the center of gravity of the target object, and the grasping vector includes a grasping posture; and Control the robot arm and hand to grasp the target object according to the grasping position and the grasping vector, and place the target object at the target position; Before receiving the command instruction, the method further includes: Scanning at least one sample object associated with a pick-and-place task performed by the mobile robot to obtain a plurality of template images corresponding to each of the at least one sample object, wherein the at least one sample object includes the target object, the plurality of template images being taken from a plurality of different viewing angles around each sample object; inserting grasping data of at least one pair of grasping points into each template image, wherein the grasping data includes a grasping position and a grasping vector associated with the at least one sample item, the at least one pair of grasping points being located at locations on each sample item that contact fingers of the hand so that each sample item can be grasped by the hand; and storing the template image and the grasping data corresponding to the template image as feature data of a sample object corresponding to the at least one sample object; The step of extracting visual feature data from the one or more images and matching the extracted visual feature data with feature data of a target object preset in a database to identify the target object includes: extracting a plurality of visual feature data from each frame of the one or more images using a directional fast rotation brief algorithm; and Matching the plurality of visual feature data with visual feature data of the target object using a balanced binary search method and a nearest Euclidean distance algorithm to identify the target object, wherein the visual feature data of the target object is extracted from a template image of the target object by a directional fast rotation brief algorithm; Wherein, determining the grasping position and grasping vector of the target object includes: When the robot arm reaches out to grab the target object, the camera is used to capture multiple images of the target object; determining a first promising region of each of the plurality of images of the target object based on the grasping position and the grasping vector inserted into the template image; Control the robot arm to move close to the first promising area, and determine a second promising area of the first promising area of each of the multiple images of the target object by performing multiple detections in the first promising area, find the best grasping position and the best grasping vector for the first promising area in each detection, and control the hand to approach the target object according to the best grasping position and the best grasping vector after each detection; The hand is controlled to move toward the target object until the robot arm grasps the target object according to the judgment result, wherein the judgment result refers to the grasping position and grasping vector detected when the distance between the hand and the target object is less than a preset distance.
Citation Information
Patent Citations
ORB key frame closed-loop detection SLAM method capable of improving consistency of position and pose of robot
CN105856230A
Synchronous localization and mapping method based on improved image matching strategy
CN108416385A
