Object grabbing method and device and self-moving cleaning equipment

Abstract

The embodiment of the invention is suitable for the technical field of intelligent household appliances, and provides an object grabbing method and device and self-moving cleaning equipment, the method is applied to the self-moving cleaning equipment, the self-moving cleaning equipment comprises a mechanical arm, and the mechanical arm is used for grabbing an object through a tail end clamping jaw. The method comprises the steps that image data and point cloud data of an object to be grabbed are collected; determining a target grabbing point of the to-be-grabbed object according to the image data and the point cloud data of the to-be-grabbed object; and the mechanical arm is controlled to grab the to-be-grabbed object based on the target grabbing point. By means of the method, the object grabbing precision of the mechanical arm can be improved.

Description

Technical Field

[0001] The present application belongs to the technical field of smart home appliances, and in particular, relates to an object grasping method, device and self-moving cleaning equipment. Background Art

[0002] The robotic arm can grab objects and be applied to cleaning equipment. The position of the object can be changed by the robotic arm, so that the bottom and surrounding of the object can be thoroughly cleaned.

[0003] However, when using a robotic arm to grasp an object, there is a problem of inaccurate grasping. For example, when there is an error in object recognition, the grasping point of the object is inaccurately calculated, or there is interference during the execution process, the robotic arm is prone to inaccurate grasping, thereby reducing cleaning efficiency. Summary of the invention

[0004] In view of this, embodiments of the present application provide a grasping method, an apparatus, a self-moving cleaning device, a medium, and a program product to improve the accuracy of a robot arm in grasping objects.

[0005] A first aspect of an embodiment of the present application provides a grasping method, which is applied to a self-moving cleaning device, wherein the self-moving cleaning device comprises a mechanical arm, and the mechanical arm is used to grasp an object through an end gripper, and the method comprises:

[0006] Collect image data and point cloud data of the object to be grasped;

[0007] Determining a target grasping point of the object to be grasped according to the image data and the point cloud data of the object to be grasped;

[0008] The robot arm is controlled to grasp the object to be grasped based on the target grasping point.

[0009] A second aspect of an embodiment of the present application provides a grasping device, which is applied to a self-moving cleaning device, wherein the self-moving cleaning device comprises a mechanical arm, wherein the mechanical arm is used to grasp an object through an end gripper, and the device comprises:

[0010] An acquisition module, used to acquire image data and point cloud data of the object to be grasped;

[0011] A determination module, used to determine a target grasping point of the object to be grasped according to the image data and point cloud data of the object to be grasped;

[0012] A control module is used to control the robot arm to grasp the object to be grasped based on the target grasping point.

[0013] A third aspect of an embodiment of the present application provides a self-moving cleaning device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0014] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0015] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a self-moving cleaning device, the self-moving cleaning device executes the method described in the first aspect.

[0016] Compared with the prior art, the embodiments of the present application have the following advantages:

[0017] The self-moving cleaning device in the embodiment of the present application has a mechanical arm, and the mechanical arm can grab objects. When grabbing an object, the self-moving cleaning device can collect image data and point cloud data, so as to determine a more accurate target grab point based on the image data and the point cloud data. After obtaining a more accurate target grab point, the target posture of the mechanical arm when the mechanical arm reaches the target grab point can be determined according to the target grab point, so as to control the mechanical arm to reach the target posture. When the mechanical arm is in the target posture, the end clamp of the mechanical arm can be closed to grab the object. When determining the target grab point, the self-moving cleaning device combines image data and point cloud data. The image data can have the contour and texture features of the object, so as to facilitate the identification and classification of the object. The point cloud data has the three-dimensional spatial features of the object, so as to facilitate the determination of the spatial shape and the positional relationship of each part. Combining different features, the target grab point of a more accurate object can be determined. The precise target grasping point enables the end gripper to better grasp the object, making it easier to grasp the object successfully, improving the grasping accuracy of the robot arm, and thus improving the cleaning efficiency of the self-moving cleaning equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0019] Figure 1 is a schematic diagram of a self-moving cleaning device provided in an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of another self-moving cleaning device provided in an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of the steps of a grabbing method provided in an embodiment of the present application;

[0022] Figure 4 It is a schematic diagram of the steps of another grabbing method provided in an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of a grasping device provided in an embodiment of the present application;

[0024] Figure 6 It is a schematic diagram of a self-moving cleaning device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the present application.

[0026] The object grabbing method in the embodiment of the present application is applied in a self-moving cleaning device. Figure 1 Schematic diagram of a self-moving cleaning device provided in an embodiment of the present application. Figure 1 As shown, the self-moving cleaning device can include a self-moving chassis and a mechanical arm, and the mechanical arm can be arranged on the self-moving chassis. The self-moving chassis can walk on the ground based on walking wheels and universal wheels.

[0027] The robot arm may include a first joint M1, a second joint M2, a third joint M3, a fourth joint M4, a fifth joint M5, and a terminal clamp. The first joint M1 and the second joint M2 may be indirectly connected, for example, the first joint and the second joint may be connected via a rotating seat. A connecting rod is connected between the second joint M2 and the third joint M3; a connecting rod is connected between the third joint M3 and the fourth joint M4; a connecting rod is connected between the fourth joint M4 and the fifth joint M5; and the fifth joint M5 is connected to the terminal clamp.

[0028] Among them, the first joint M1 is connected to the chassis and is used to control the rotation of the robotic arm around the vertical axis; the second joint M2 supports the robotic arm through pitch rotation; the third joint M3 is in the same direction as the joints of the second joint M2 and the fourth joint M4; the fifth joint M5 is the self-rotating joint of the end gripper, as shown in the figure, the fifth joint M5 can self-rotate along the axis where the connecting rod between the fourth joint M4 and the fifth joint M5 is located, so as to adjust the direction of the end gripper to grip the object; the end gripper can place or grip the object by opening and closing.

[0029] like Figure 1 As shown, a time of flight (TOF) sensor and a camera are installed at the front end of the chassis of the self-propelled cleaning device, which can be used to collect point cloud data and image data. The TOF sensor can calculate the depth information of each point on the surface of an object by measuring the time from the emission to the reception of the light signal, where the depth information can be the distance between each point on the surface of the object and a reference position. Point cloud data can be constructed based on the depth information of each point on the surface of the object. In addition, the end of the robotic arm can also have a camera for collecting image data. Figure 2 As shown, Figure 2 Schematic diagram of another self-moving cleaning device provided in an embodiment of the present application; Figure 2 As shown, the end gripper is equipped with an end camera and an end fill light, and clear image data can also be collected based on the end camera and the end fill light.

[0030] During the movement of the self-mobile cleaning device, the self-mobile cleaning device can collect point cloud data through the TOF sensor and image data through the camera. Based on the collected point cloud data and image data, the self-mobile cleaning device can determine whether there is an obstacle in front. When there is an obstacle in front, the self-mobile cleaning device can determine whether the obstacle is an object to be grasped. If the obstacle is an object to be grasped, the object can be grasped by the robotic arm, thereby changing the position of the object.

[0031] The grasping process of a robotic arm is a complex multi-stage process. During the grasping process of an object, there may be problems such as target recognition error, unstable grasping point calculation, and more interference in the execution process, which may lead to empty grasping or unstable object grasping. Based on this, an embodiment of the present application provides a grasping method for improving the grasping accuracy of a robotic arm.

[0032] The technical solution of the present application is described below through specific embodiments.

[0033] Reference Figure 3 , shows a schematic diagram of the step flow of a grabbing method provided in an embodiment of the present application, which may specifically include the following steps:

[0034] S301, collecting image data and point cloud data of the object to be grasped.

[0035] The execution subject of this embodiment is the self-moving cleaning device mentioned above. The image data mentioned above can be acquired by a camera, and the point cloud data can be acquired by a TOF sensor.

[0036] S302: Determine a target grasping point of the object to be grasped according to the image data and the point cloud data of the object to be grasped.

[0037] According to the image data, the self-moving cleaning device can identify the category of the object to be grasped. In a possible implementation, the self-moving cleaning device can deploy an image recognition algorithm, and based on the image recognition algorithm, the object category can be identified.

[0038] According to the category of the object to be grasped, the self-moving cleaning device can determine the corresponding grasping mode. In a possible implementation, the self-moving cleaning device can store grasping modes associated with the object category. For example, if the object to be grasped is a slipper, the target grasping point can be located on the upper of the slipper, and the grasping mode can be grasping the upper; if the object to be grasped is a high-heeled shoe, the target grasping point can be located on the heel, and the grasping mode can be grasping the heel.

[0039] Based on the grasping mode, the self-mobile cleaning device can determine a set of candidate grasping points. The grasping mode may include grasping area information, and the set of candidate grasping points can be determined from the grasping area. The set of candidate grasping points can be a set of all point clouds in the grasping area, or a set of some points in the edge area of ​​the grasping area. For example, if the object to be grasped is a slipper, and the grasping mode is to grasp the upper, the self-mobile cleaning device can determine the upper area of ​​the slipper to be grasped, and each point in the upper area can be a candidate grasping point; if the object to be grasped is a high-heeled shoe, and the grasping mode is to grasp the heel, the self-mobile cleaning device can determine the heel area of ​​the slipper to be grasped, and each point in the heel area can be a candidate grasping point.

[0040] Based on the point cloud data, the self-moving cleaning device can determine the target grasping point from the set of candidate grasping points. When determining the target grasping point from the set of candidate grasping points, the target grasping point can be determined from multiple candidate grasping points based on the scores and attribute information of the multiple candidate grasping points. Among them, the attribute information includes reachable or unreachable. The attribute information of reachable is used to characterize whether the robot arm can reach the candidate grasping point, and the attribute information of unreachable is used to characterize whether the robot arm can reach the candidate grasping point. That is, the target grasping point can be determined based on the score of the selected grasping point and whether the robot arm can reach the candidate grasping point.

[0041] In one possible implementation, when determining the target grasping point from the set of candidate grasping points, each candidate grasping point can be scored, and then a preset number of candidate grasping points with higher scores are selected, and whether the robot arm can reach the candidate grasping point is determined in descending order of scores. The candidate grasping point with the highest score that the robot arm can reach is used as the target grasping point. The grasping point with the highest score is selected to maximize the success rate and stability of grasping. Because the scoring model has comprehensively considered the impact of multiple factors on grasping, a grasping point with a high score usually means that in the current situation, using the end gripper for grasping can achieve better results. For example, a grasping point with a high score may be able to provide a larger contact area and a more reasonable force distribution, thereby reducing the risk of the object slipping or being damaged during the grasping process.

[0042] In another possible implementation, when determining the target grasping point from the candidate grasping point set, it is possible to first determine whether the robot arm can reach each candidate grasping point, and then delete the candidate grasping points that the robot arm cannot reach from the candidate grasping point set to update the candidate grasping point set. After the candidate grasping point set is updated, each candidate grasping point can be scored, and then a preset number of candidate grasping points with higher scores can be selected.

[0043] In another possible implementation, when determining the target grasping point from the set of candidate grasping points, each candidate grasping point may be scored. When the score of any candidate grasping point is greater than a preset score, it may be determined whether the robot arm can reach the candidate grasping point; if the robot arm can reach the candidate grasping point, the candidate grasping point may be directly used as the target grasping point; if the robot arm cannot reach the candidate grasping point, each candidate grasping point may continue to be scored until the target grasping point is determined. The target grasping point is determined as a candidate grasping point that the robot arm can reach and the score is greater than the preset score, which can reduce the amount of calculation during the grasping process.

[0044] When scoring, candidate point clouds corresponding to multiple candidate grasping points can be determined respectively according to the point cloud data. As an example, the candidate point cloud can be a point cloud within a preset range near the candidate grasping point. For example, a point cloud composed of spatial points within a circle with a preset length as the radius and the candidate grasping point as the center can be used as a candidate point cloud.

[0045] The candidate point cloud has geometric features, and the geometric features of the candidate point cloud can be directly determined based on the contour of the candidate point cloud. The geometric features of the candidate point cloud can also be represented by a feature vector. Among them, the feature vector may include surface normal vectors, curvature information, depth gradients, and neighborhood point distribution. The method of extracting feature vectors from point clouds can refer to the prior art and will not be repeated here.

[0046] According to the geometric features of the candidate point cloud, the scores corresponding to multiple candidate grasping points can be determined respectively. In a possible implementation, the feature vector is input into the scoring model, and the score corresponding to each feature vector can be calculated. The scoring model is a pre-trained deep learning model. Among them, the scoring model is constructed based on a large amount of experimental data and machine learning algorithms, and comprehensively considers the influence of multiple factors on the grasping success rate. For example, the model will consider the relationship between the position of the grasping point and the center of gravity of the object. A grasping point close to the center of gravity of the object can usually provide a more stable grasping effect; the model will also consider the contact mode between the gripper and the grasping point. Different contact modes will affect the magnitude and distribution of friction, thereby affecting the firmness of the grasping. In addition, the model will also consider the material properties of the object, such as the smoothness and hardness of the surface, which will also have an important impact on the grasping. Through the scoring model, the device can quantitatively evaluate each candidate grasping point and provide a scientific basis for subsequent decision-making. For example, the computer device can deploy a tool for scoring, Grasp Pose Detection (GPD), based on which, the scoring of each point to be grasped can be achieved. The specific scoring process can refer to the prior art and will not be repeated here. In a possible implementation, a higher score may be given to candidate feature points located in the edge region of the object.

[0047] In another possible implementation, a scoring table may be searched to determine the score corresponding to each spatial contour, thereby obtaining the score of the candidate point cloud.

[0048] Due to the limited structure and range of motion of the device, not all candidate grasping points can be reached by the end gripper. Therefore, when selecting from multiple candidate grasping points, the device will determine whether each candidate grasping point is within the reachable range of the end gripper based on its own kinematic model and current position and posture. Attribute information can characterize whether the candidate grasping point is within the reachable range of the end gripper. As an example, the distance between the candidate grasping point and the base of the robot arm can be determined; if the distance is greater than the maximum active radius of the robot arm, the attribute information can be determined to be unreachable. As another example, the joint angles of each joint of the robot arm when reaching the candidate grasping point can be calculated by the inverse kinematics (IK) algorithm; if the joint angle has no solution, the attribute information is determined to be unreachable. If the distance between the candidate grasping point and the base of the robot arm is greater than the maximum active radius of the robot arm, and the joint angle has a solution, it is determined that the robot arm can reach the candidate grasping point. The specific calculation process of the inverse kinematics algorithm can refer to the prior art and will not be repeated here.

[0049] S303: Control the robot arm to grasp the object to be grasped based on the target grasping point.

[0050] According to the inverse kinematics algorithm, the target joint angle of the designated joint of the robot arm can be calculated when the end gripper of the robot arm reaches the target grasping point; wherein the designated joint can be the joint that controls the end posture, and the designated joint can be the second joint, the third joint and the fourth joint mentioned above.

[0051] When grasping an object, the end gripper is opened and closed to grasp it. The two grippers of the end gripper need to be located at the specified positions. For example, when grasping the upper area of ​​a slipper, the two grippers need to be located on both sides of the upper to achieve grasping. Therefore, the direction in which the end gripper opens also needs to be determined.

[0052] The direction in which the end jaw opens can be determined by the rotation joint of the end jaw, so the rotation joint angle of the rotation joint of the end jaw can be determined when the specified joint is at the target joint angle. The rotation joint can be the fifth joint mentioned above. The end jaw direction can be determined by a deep learning algorithm. After determining the end jaw direction, the rotation joint angle can be determined according to the end jaw direction.

[0053] Based on the target joint angle and the current joint angle, the designated joint of the robot arm can be controlled, and based on the rotational joint angle and the current angle of the rotational joint, the rotational joint of the robot arm can be controlled, so that the designated joint is at the target joint angle and the rotational joint is at the rotational joint angle. When the designated joint is at the target joint angle and the rotational joint is at the rotational joint angle, the end gripper reaches the target grasping point with the specified posture, thereby opening the gripper and closing it, so as to grasp the object at the target grasping point through the end gripper.

[0054] In this embodiment, the self-propelled cleaning device makes full use of advanced sensor technology, machine learning algorithms and the device's own kinematic model when determining the target grasping point. By comprehensively considering the geometric features of the object, the evaluation results of the scoring model and the accessibility of the end gripper, the device can accurately select the best target grasping point, thereby achieving efficient and reliable grasping operations, providing a strong guarantee for completing the cleaning task.

[0055] In order to better illustrate the solution of this application, the solution in this embodiment is described below with reference to the following specific embodiments. Figure 4 , showing a schematic diagram of the step flow of another grasping method provided in an embodiment of the present application.

[0056] During the movement of the self-mobile cleaning device, RGB images and TOF point cloud data can be collected. The RGB image is the above image data, and the TOF point cloud data is the above point cloud data.

[0057] Based on the collected RGB images and TOF point cloud data, the self-propelled cleaning device can identify obstacles. When an obstacle is identified, the category of the obstacle can be identified, and then it can be determined whether it is an object to be grasped according to the category of the object. If the obstacle is an object to be grasped, the target grasping point is determined based on the RGB image and TOF point cloud data.

[0058] In one possible implementation, the self-moving cleaning device can analyze the RGB image and point cloud data of a target area where the object to be grasped is located to determine multiple candidate grasping points. And determine the target grasping point from multiple candidate grasping points. First, by analyzing the RGB image and ToF point cloud data of the target area, a candidate set of multiple grasping points is generated. These candidate points are screened based on the appearance features and point cloud data of the object, and further identify the category, orientation and other posture information of the target object through the image classification model. The spatial coordinates of the grasping point are finalized at this stage and returned in the form of Grasp ID, which is used to identify the specific location and grasping mode of the grasping point. Grasp ID is a unique identifier for the grasping point.

[0059] The calculation of the grasping posture is based on the category and posture information of the target object. Depending on the type of target object, such as slippers, ordinary shoes or boots, the system adopts the corresponding grasping mode and further adjusts the position of the grasping point based on the ToF data. Through the inverse kinematics calculation of the robot arm, the target angle of the robot arm joint is obtained, so that the end effector can accurately reach the grasping point position. Especially for objects with complex shapes, the system will dynamically adjust the grasping posture based on the spatial posture of the object, thereby improving the stability and accuracy of the grasping action.

[0060] The approach angle is calculated by comparing the grasping direction of the object with the current direction of the robot arm. The grasping direction of the object is provided by the classification model, and the current direction of the robot arm relative to the object is calculated by the sensor. The difference in direction between the two is normalized to generate the gripper rotation angle required for the robot arm to approach the object. This angle is crucial in actual operation because it directly determines whether the end of the robot arm can contact the target object in the best posture to complete the grasping action.

[0061] After the grasping posture and approach angle are determined, the robot arm starts to perform the grasping task. The grasping task includes the robot arm moving to the target position according to the plan and closing the gripper to complete the grasping operation. The robot arm then lifts the object and transfers it to the predefined carrying position to ensure the stability of the object after grasping. During the entire grasping process, the system monitors the status of the robot arm and the feedback information of the target object in real time. If a grasping failure or object falling off is detected, the compensation mechanism will be immediately triggered to re-plan the grasping action.

[0062] Through the above technical solution, the present application realizes a full-process automated grasping system from target recognition to action execution, which significantly improves the success rate of grasping tasks and the adaptability of the system.

[0063] It involves everything from visual perception of objects to motion planning and then to the final execution. The visual perception part uses a multimodal sensor to fuse RGB images and ToF point cloud data, identify the category of the target object, calculate the spatial coordinates of the optimal grasping point, and combine the posture information of the target object to provide accurate data support for subsequent motion planning. The motion planning part calculates the grasping posture and approach angle of the end effector of the robot arm through an inverse kinematics algorithm based on the spatial position of the grasping point, combined with the type and posture of the object. The execution part converts the grasping posture and approach angle into joint control instructions for the robot arm, driving the robot arm to accurately reach its position and complete the grasping task.

[0064] The purpose of this application is to solve the problems existing in the existing grasping system, such as target recognition error, unstable grasping point calculation and many interferences in the execution process, so as to achieve efficient and reliable grasping of diverse target objects and improve the grasping accuracy and adaptability of the overall system.

[0065] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0066] Reference Figure 5 , shows a schematic diagram of a grasping device provided in an embodiment of the present application, the device is applied to a self-moving cleaning device, the self-moving cleaning device includes a mechanical arm, the mechanical arm is used to grasp an object through an end gripper, the device may specifically include a collection module 51, a determination module 52 and a control module 53, wherein:

[0067] The acquisition module 51 is used to acquire image data and point cloud data of the object to be grasped;

[0068] A determination module 52, configured to determine a target grasping point of the object to be grasped according to the image data and the point cloud data of the object to be grasped;

[0069] The control module 53 is used to control the robot arm to grasp the object to be grasped based on the target grasping point.

[0070] In a possible implementation, the determining module includes:

[0071] An identification submodule, used to identify the category of the object to be grasped according to the image data;

[0072] A grasping mode determination submodule, used to determine a corresponding grasping mode according to the category of the object to be grasped;

[0073] A candidate grasping point set determination submodule, used to determine a candidate grasping point set based on the grasping mode;

[0074] The target grasping point determination submodule is used to determine the target grasping point from the candidate grasping point set based on the point cloud data.

[0075] In a possible implementation, the candidate grasping point set includes a plurality of candidate grasping points, and the target grasping point determination submodule includes:

[0076] A candidate point cloud determination unit, used to determine candidate point clouds corresponding to a plurality of candidate grasping points respectively according to the point cloud data;

[0077] A scoring unit, used to determine scores corresponding to the plurality of candidate grasping points respectively according to the geometric features of the candidate point cloud;

[0078] An attribute information determination unit, used to respectively determine the attribute information corresponding to the plurality of candidate grasping points; wherein the attribute information includes reachable or unreachable, and the attribute information being reachable is used to indicate that the manipulator can reach the candidate grasping point, and the attribute information being unreachable is used to indicate that the manipulator cannot reach the candidate grasping point;

[0079] A target grasping point determining unit is used to determine the target grasping point from the plurality of candidate grasping points according to the scores and the attribute information of the plurality of candidate grasping points.

[0080] In a possible implementation manner, the attribute information determining unit includes:

[0081] A first judgment subunit is used to determine the distance between the candidate grasping point and the base of the robotic arm; if the distance is greater than the maximum activity radius of the robotic arm, the attribute information of the candidate grasping point is determined to be unreachable;

[0082] The second judgment subunit is used to solve the joint angles of each joint of the robot arm when reaching the candidate grasping point through an inverse kinematics algorithm; if the joint angles cannot be obtained through the inverse kinematics algorithm, the attribute information corresponding to the candidate grasping point is determined to be unreachable.

[0083] In a possible implementation, the scoring unit includes:

[0084] A feature extraction subunit, used to extract a feature vector corresponding to each of the candidate point clouds, wherein the feature vector is used to characterize the geometric features of the candidate point cloud;

[0085] The scoring subunit is used to calculate the score corresponding to each of the feature vectors as the score of each of the corresponding candidate grasping points through a scoring model, and the scoring model is a pre-trained deep learning model.

[0086] In a possible implementation, the control module includes:

[0087] A target joint angle determination submodule, used to determine a target joint angle of a designated joint of the robotic arm when the end gripper of the robotic arm reaches the target grasping point;

[0088] A rotation joint angle determination submodule, used to determine the rotation joint angle of the rotation joint of the end gripper when the designated joint is at the target joint angle; wherein the rotation joint is used to control the rotation of the end gripper;

[0089] The grasping submodule is used to control the designated joint to be at the target joint angle and the rotary joint to be at the rotary joint angle, so as to grasp the object at the target grasping point through the end gripper.

[0090] In a possible implementation, the mechanical arm includes a first joint, a second joint, a third joint, a fourth joint, a fifth joint and an end gripper;

[0091] The designated joints are the second joint, the third joint and the fourth joint; the rotational joint is the fifth joint;

[0092] The first joint and the second joint are connected;

[0093] A connecting rod is connected between the second joint and the third joint;

[0094] A connecting rod is connected between the third joint and the fourth joint;

[0095] A connecting rod is connected between the fourth joint and the fifth joint;

[0096] The fifth joint is connected to the end clamp;

[0097] Among them, the first joint is connected to the chassis and is used to control the rotation of the robotic arm around the vertical axis; the second joint supports the robotic arm through pitch rotation; the third joint is in the same direction as the second joint and the fourth joint; the fifth joint is the self-rotating joint of the end clamp, which is used to adjust the direction of the end clamp to clamp the object; the end clamp is used to clamp or place the object by opening and closing.

[0098] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.

[0099] Figure 6 This is a schematic diagram of the structure of a self-moving cleaning device provided in an embodiment of the present application. Figure 6 As shown, the self-moving cleaning device 600 of this embodiment includes: at least one processor 60 ( Figure 6 Only one is shown in the figure), a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, wherein the processor 60 implements the steps of any of the above-mentioned method embodiments when executing the computer program 62.

[0100] The self-moving cleaning device 600 may be a household robot. The self-moving cleaning device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will appreciate that Figure 6 It is only an example of the self-moving cleaning device 600 and does not constitute a limitation on the self-moving cleaning device 600. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0101] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0102] The memory 61 can be the internal storage unit of the self-mobile cleaning device 600 in some embodiments, such as the hard disk or memory of the self-mobile cleaning device 600. The memory 61 can also be the external storage device of the self-mobile cleaning device 600 in other embodiments, such as the plug-in hard disk equipped on the self-mobile cleaning device 600, the smart memory card (Smart Media Card, SMC), the secure digital (Secure Digital, SD) card, the flash card (FlashCard) etc. Further, the memory 61 can also include both the internal storage unit of the self-mobile cleaning device 600 and the external storage device. The memory 61 is used to store operating systems, applications, boot loaders (BootLoader), data and other programs, such as program codes of the computer programs, etc. The memory 61 can also be used to temporarily store data that has been output or is about to be output.

[0103] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0104] An embodiment of the present application provides a computer program product. When the computer program product is run on a self-moving cleaning device, the self-moving cleaning device can implement the steps in the above-mentioned various method embodiments when executing the computer program product.

[0105] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for grasping an object, characterized in that: Applied to a self-moving cleaning device, the self-moving cleaning device includes a mechanical arm, the mechanical arm is used to grasp an object, the method includes: Collect image data and point cloud data of the object to be grasped; Determining a target grasping point of the object to be grasped according to the image data and the point cloud data of the object to be grasped; The robot arm is controlled to grasp the object to be grasped based on the target grasping point.

2. The method according to claim 1, characterized in that Determining a target grasping point of the object to be grasped according to the image data and the point cloud data of the object to be grasped includes: According to the image data, identifying the category of the object to be grasped; Determining a corresponding grasping mode according to the category of the object to be grasped; Based on the grasping mode, determining a set of candidate grasping points; Based on the point cloud data, the target grasping point is determined from the set of candidate grasping points.

3. The method according to claim 2, characterized in that The candidate grasping point set includes a plurality of candidate grasping points, and determining the target grasping point from the candidate grasping point set based on the point cloud data includes: Determine, according to the point cloud data, candidate point clouds corresponding to the plurality of candidate grasping points; Determining scores corresponding to the plurality of candidate grasping points respectively according to the geometric features of the candidate point cloud; Determine attribute information corresponding to the plurality of candidate grasping points respectively; wherein the attribute information includes reachable or unreachable, the attribute information being reachable is used to indicate that the robot arm can reach the candidate grasping point, and the attribute information being unreachable is used to indicate that the robot arm cannot reach the candidate grasping point; The target grasping point is determined from the plurality of candidate grasping points according to the scores and the attribute information of the plurality of candidate grasping points.

4. The method according to claim 3, characterized in that The step of respectively determining the attribute information corresponding to the plurality of candidate grasping points comprises: Determine the distance between the candidate grasping point and the base of the robotic arm; if the distance is greater than the maximum activity radius of the robotic arm, determine that the attribute information of the candidate grasping point is unreachable; The joint angles of the joints of the robot arm when reaching the candidate grasping point are solved by the inverse kinematics algorithm; if the joint angles cannot be solved by the inverse kinematics algorithm, the attribute information of the corresponding candidate grasping point is determined to be unreachable.

5. The method according to claim 3, characterized in that Determining scores corresponding to the plurality of candidate grasping points respectively according to the geometric features of the candidate point cloud includes: Extracting a feature vector corresponding to each of the candidate point clouds, wherein the feature vector is used to characterize the geometric features of the candidate point cloud; The score corresponding to each of the feature vectors is calculated as the score of each of the corresponding candidate grasping points through a scoring model, and the scoring model is a pre-trained deep learning model.

6. The method according to any one of claims 1 to 5, characterized in that: The controlling the robot arm to grasp the object to be grasped based on the target grasping point includes: Determining a target joint angle of a designated joint of the robotic arm when the end gripper of the robotic arm reaches the target grasping point; Determining a rotational joint angle of a rotational joint of the end gripper when the designated joint is at the target joint angle; wherein the rotational joint is used to control the opening direction of the end gripper; The designated joint is controlled to be at the target joint angle and the rotary joint is controlled to be at the rotary joint angle, so as to grasp the object to be grasped at the target grasping point through the end gripper.

7. The method according to claim 6, characterized in that The mechanical arm comprises a first joint, a second joint, a third joint, a fourth joint, a fifth joint and an end gripper; The designated joints are the second joint, the third joint and the fourth joint; the rotational joint is the fifth joint; The first joint and the second joint are connected; A connecting rod is connected between the second joint and the third joint; A connecting rod is connected between the third joint and the fourth joint; A connecting rod is connected between the fourth joint and the fifth joint; The fifth joint is connected to the end clamp; Among them, the first joint is connected to the chassis and is used to control the rotation of the robotic arm around the vertical axis; the second joint supports the robotic arm through pitch rotation; the third joint is in the same direction as the second joint and the fourth joint; the fifth joint is the self-rotating joint of the end clamp, which is used to adjust the direction of the end clamp to clamp the object; the end clamp is used to clamp or place the object by opening and closing.

8. An object grasping device, characterized in that: Applied to a self-moving cleaning device, the self-moving cleaning device includes a mechanical arm, the mechanical arm is used to grab an object, and the device includes: An acquisition module, used to acquire image data and point cloud data of the object to be grasped; A determination module, used to determine a target grasping point of the object to be grasped according to the image data and point cloud data of the object to be grasped; A control module is used to control the robot arm to grasp the object to be grasped based on the target grasping point.

9. The device according to claim 8, characterized in that The determining module comprises: An identification submodule, used to identify the category of the object to be grasped according to the image data; A grasping mode determination submodule, used to determine a corresponding grasping mode according to the category of the object to be grasped; A candidate grasping point set determination submodule, used to determine a candidate grasping point set based on the grasping mode; The target grasping point determination submodule is used to determine the target grasping point from the candidate grasping point set based on the point cloud data.

10. The device according to claim 9, characterized in that The candidate grasping point set includes a plurality of candidate grasping points, and the target grasping point determination submodule includes: A candidate point cloud determination unit, used to determine candidate point clouds corresponding to a plurality of candidate grasping points respectively according to the point cloud data; A scoring unit, used to determine scores corresponding to the plurality of candidate grasping points respectively according to the geometric features of the candidate point cloud; An attribute information determination unit, used to respectively determine the attribute information corresponding to the plurality of candidate grasping points; wherein the attribute information includes reachable or unreachable, and the attribute information being reachable is used to indicate that the manipulator can reach the candidate grasping point, and the attribute information being unreachable is used to indicate that the manipulator cannot reach the candidate grasping point; A target grasping point determining unit is used to determine the target grasping point from the plurality of candidate grasping points according to the scores and the attribute information of the plurality of candidate grasping points.

11. The device according to claim 10, characterized in that The attribute information determining unit includes: A first judgment subunit is used to determine the distance between the candidate grasping point and the base of the robotic arm; if the distance is greater than the maximum activity radius of the robotic arm, the attribute information of the candidate grasping point is determined to be unreachable; The second judgment subunit is used to solve the joint angles of each joint of the robot arm when reaching the candidate grasping point through an inverse kinematics algorithm; if the joint angles cannot be obtained through the inverse kinematics algorithm, the attribute information corresponding to the candidate grasping point is determined to be unreachable.

12. The device according to claim 10, characterized in that The scoring unit comprises: A feature extraction subunit, used to extract a feature vector corresponding to each of the candidate point clouds, wherein the feature vector is used to characterize the geometric features of the candidate point cloud; The scoring subunit is used to calculate the score corresponding to each of the feature vectors as the score of each of the corresponding candidate grasping points through a scoring model, and the scoring model is a pre-trained deep learning model.

13. The device according to any one of claims 8 to 12, characterized in that: The control module comprises: A target joint angle determination submodule, used to determine a target joint angle of a designated joint of the robotic arm when the end gripper of the robotic arm reaches the target grasping point; A rotation joint angle determination submodule, used to determine the rotation joint angle of the rotation joint of the end gripper when the designated joint is at the target joint angle; wherein the rotation joint is used to control the rotation of the end gripper; The grasping submodule is used to control the designated joint to be at the target joint angle and the rotary joint to be at the rotary joint angle, so as to grasp the object to be grasped at the target grasping point through the end clamp.

14. The device according to claim 13, characterized in that The mechanical arm comprises a first joint, a second joint, a third joint, a fourth joint, a fifth joint and an end gripper; The designated joints are the second joint, the third joint and the fourth joint; the rotational joint is the fifth joint; The first joint and the second joint are connected; A connecting rod is connected between the second joint and the third joint; A connecting rod is connected between the third joint and the fourth joint; A connecting rod is connected between the fourth joint and the fifth joint; The fifth joint is connected to the end clamp; Among them, the first joint is connected to the chassis and is used to control the rotation of the robotic arm around the vertical axis; the second joint supports the robotic arm through pitch rotation; the third joint is in the same direction as the second joint and the fourth joint; the fifth joint is the self-rotating joint of the end clamp, which is used to adjust the direction of the end clamp to clamp the object; the end clamp is used to clamp or place the object by opening and closing.

15. A self-moving cleaning device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The self-moving cleaning device comprises a robotic arm, and the robotic arm is used to grab an object. When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

16. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

17. A computer program product, characterized in that When the computer program product is run on a self-moving cleaning device, the self-moving cleaning device is enabled to perform the method according to any one of claims 1 to 7.

Images