Grabbing control method and device of robot

Abstract

The invention provides a grabbing control method and device of a robot, the robot, a storage medium and a program product. A first collecting assembly is controlled to collect first sensing data in the current environment where the robot is located, and a mechanical arm is controlled to drive a second collecting assembly to collect second sensing data in the current environment; and controlling the mechanical arm to grab the target object based on the first sensing data and the second sensing data. Thus, the target object can be grabbed according to the first sensing data and the second sensing data obtained by the first collection assembly and the second collection assembly which are different in position, the target object can be grabbed according to data of different view angles, the possibility of misrecognition caused by limited obtained information under a single view angle is reduced, and the recognition accuracy is improved. And the accuracy and the flexibility of the grabbing control method are improved.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of robotics technology, and in particular to a robot grasping control method and device, a robot, a storage medium and a program product. Background Art

[0002] Current robots only place cameras on their bodies or on their robotic arms to determine which objects can be grasped. For objects of different categories, the judgment results are inaccurate due to the single judgment angle, resulting in low accuracy and poor flexibility. Summary of the invention

[0003] In view of this, the embodiments of the present application at least provide a robot grasping control method and device, a robot, a storage medium and a program product.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The embodiment of the present application provides a grasping control method of a robot, wherein the robot includes a body, a mechanical arm arranged on the top of the body, a first acquisition component arranged on the body, and a second acquisition component arranged on the mechanical arm, and the method includes:

[0006] Controlling the first acquisition component to collect first perception data in the current environment of the robot;

[0007] Control the robotic arm to drive the second acquisition component to collect second perception data in the current environment;

[0008] Based on the first perception data and the second perception data, the robotic arm is controlled to grasp a target object that can be grasped in the current environment.

[0009] In some embodiments, the robotic arm is controlled to drive the second acquisition component to collect second perception data in the current environment, including: based on the first perception data, identifying whether there is a graspable target object in the current environment to obtain a recognition result; when the recognition result indicates that there is a target object, the robotic arm is controlled to drive the second acquisition component to collect information on the target object to obtain second perception data.

[0010] In some embodiments, based on the first perception data and the second perception data, the robotic arm is controlled to grasp a target object that can be grasped in the current environment, including: determining a first recognition result corresponding to the first perception data and a second recognition result corresponding to the second perception data; based on the first recognition result and the second recognition result, the robotic arm is controlled to grasp the target object.

[0011] In some embodiments, the first recognition result includes a first target category corresponding to the target object, and the second recognition result includes a second target category corresponding to the target object; based on the first recognition result and the second recognition result, the robotic arm is controlled to grasp the target object, including: when the first target category and the second target category are the same, the robotic arm is controlled to grasp the target object based on the first target grasping point corresponding to the target object; the first target grasping point is identified from the first perception data and / or the second perception data based on the first target category; or, when the first target category and the second target category are different, a first target strategy to be executed is determined to control the robotic arm to grasp the target object based on the first target strategy.

[0012] In some embodiments, controlling a robotic arm to grasp a target object based on a first target strategy includes one of the following: controlling the robotic arm to give up grasping the target object; controlling the robotic arm to grasp the target object based on a first target grasping point; the first target grasping point is obtained by identifying the first target category from the first perception data and / or the second perception data; controlling the robotic arm to grasp the target object based on a second target grasping point; the second target grasping point is obtained by identifying the second target category from the first perception data and / or the second perception data; controlling the robotic arm and / or the body to detour from the current first position to the second position of the target object, controlling the first acquisition component at the second position to collect information on the target object to obtain third perception data, and controlling the robotic arm to drive the second acquisition component to collect information on the target object to obtain fourth perception data, so as to control the robotic arm to grasp the target object based on the third perception data and the fourth perception data.

[0013] In some embodiments, based on the third perception data and the fourth perception data, the robotic arm is controlled to grasp the target object, including: when the third recognition result corresponding to the third perception data and the fourth recognition result corresponding to the fourth perception data are the same, the robotic arm is controlled to grasp the target object based on the third target grasping point or the fourth target grasping point; the third target grasping point is identified from the third perception data and / or the fourth perception data based on the third target category in the third recognition result, and the fourth target grasping point is identified from the third perception data and / or the fourth perception data based on the fourth target category in the fourth recognition result; or, when the third recognition result and the fourth recognition result are different and the number of times the first acquisition component or the second acquisition component collects information on the target object exceeds a number threshold, the robotic arm is controlled to grasp the target object based on the second target strategy, and the second target strategy includes one of the following: controlling the robotic arm to give up grasping the target object; controlling the robotic arm to grasp the target object based on the third target grasping point; controlling the robotic arm to grasp the target object based on the fourth target grasping point.

[0014] In some embodiments, the robot arm is controlled to drive the second acquisition component to collect information on the target object to obtain second perception data, including: based on the first candidate perception data currently collected by the second acquisition component, identifying whether the target object meets the target condition; the target condition includes at least one of the following: the proportion of target perception data corresponding to the target object in the first candidate perception data is greater than or equal to a preset ratio, and the target object is located in the visual cone corresponding to the second acquisition component; when the target object meets the target condition, the first candidate perception data is used as the second perception data.

[0015] In some embodiments, the method also includes: when the target object does not meet the target conditions, controlling the robotic arm and / or the fuselage to drive the second acquisition component to move and collect information on the target object to obtain second candidate perception data, so as to identify whether the target object meets the target conditions based on the second candidate perception data.

[0016] In some embodiments, the robotic arm is installed in a storage bin on top of the fuselage; the first acquisition component is controlled to collect first perception data in the current environment of the robot, including: when the robotic arm is retracted into the storage bin, the first acquisition component is controlled to collect first perception data in the current environment of the robot; the robotic arm is controlled to drive the second acquisition component to collect information on the target object to obtain second perception data, including: controlling the robotic arm to perform an out-of-bin operation, and controlling the robotic arm to drive the second acquisition component to collect information on the target object to obtain second perception data.

[0017] In some embodiments, when the robotic arm is retracted into the accommodating bin, the first acquisition component is controlled to collect first perception data in the current environment of the robot, including: when the robotic arm is retracted into the accommodating bin, the robot is controlled to perform a cleaning task; while the robot is performing the cleaning task, the first acquisition component is controlled to collect first perception data in the current environment of the robot.

[0018] The embodiment of the present application provides a grasping control device of a robot, the robot comprising a body, a mechanical arm arranged on the top of the body, a first acquisition component arranged on the body, and a second acquisition component arranged on the mechanical arm, the device comprising:

[0019] A first control module, used to control the first acquisition component to collect first perception data in the current environment of the robot;

[0020] A second control module is used to control the robotic arm to drive the second acquisition component to collect second perception data in the current environment;

[0021] The third control module is used to control the robotic arm to grasp a graspable target object in the current environment based on the first perception data and the second perception data.

[0022] In some embodiments, the device also includes: an identification module, which is used to identify whether there is a graspable target object in the current environment based on the first perception data, and obtain an identification result; a second control module, which is also used to control the robotic arm to drive the second acquisition component to collect information on the target object when the recognition result indicates that there is a target object, and obtain second perception data.

[0023] In some embodiments, the third control module is also used to determine a first recognition result corresponding to the first perception data and a second recognition result corresponding to the second perception data; based on the first recognition result and the second recognition result, the robotic arm is controlled to grasp the target object.

[0024] In some embodiments, the first recognition result includes a first target category corresponding to the target object, and the second recognition result includes a second target category corresponding to the target object; the third control module is also used to control the robotic arm to grasp the target object based on the first target grasping point corresponding to the target object when the first target category and the second target category are the same; the first target grasping point is identified from the first perception data and / or the second perception data based on the first target category; or, when the first target category and the second target category are different, determine the first target strategy to be executed to control the robotic arm to grasp the target object based on the first target strategy.

[0025] In some embodiments, the third control module is also used to control the robotic arm to give up grasping the target object; control the robotic arm to grasp the target object based on the first target grasping point; the first target grasping point is obtained by identifying the first target category from the first perception data and / or the second perception data; control the robotic arm to grasp the target object based on the second target grasping point; the second target grasping point is obtained by identifying the second target category from the first perception data and / or the second perception data; control the robotic arm and / or the body to detour from the current first position to the second position of the target object, control the first acquisition component at the second position to collect information on the target object to obtain third perception data, and control the robotic arm to drive the second acquisition component to collect information on the target object to obtain fourth perception data, so as to control the robotic arm to grasp the target object based on the third perception data and the fourth perception data.

[0026] In some embodiments, the third control module is also used to control the robotic arm to grasp the target object based on the third target grasping point or the fourth target grasping point when the third recognition result corresponding to the third perception data and the fourth recognition result corresponding to the fourth perception data are the same; the third target grasping point is identified from the third perception data and / or the fourth perception data based on the third target category in the third recognition result, and the fourth target grasping point is identified from the third perception data and / or the fourth perception data based on the fourth target category in the fourth recognition result; or, when the third recognition result and the fourth recognition result are different and the number of times the first acquisition component or the second acquisition component collects information on the target object exceeds a number threshold, control the robotic arm to grasp the target object based on the second target strategy, and the second target strategy includes one of the following: controlling the robotic arm to give up grasping the target object; controlling the robotic arm to grasp the target object based on the third target grasping point; controlling the robotic arm to grasp the target object based on the fourth target grasping point.

[0027] In some embodiments, the second control module is also used to identify whether the target object meets the target condition based on the first candidate perception data currently collected by the second acquisition component; the target condition includes at least one of the following: the proportion of the target perception data corresponding to the target object in the first candidate perception data is greater than or equal to a preset ratio, and the target object is located in the visual cone corresponding to the second acquisition component; when the target object meets the target condition, the first candidate perception data is used as the second perception data.

[0028] In some embodiments, the second control module is also used to control the robotic arm and / or the fuselage to drive the second acquisition component to move and collect information on the target object when the target object does not meet the target conditions, so as to obtain second candidate perception data, so as to identify whether the target object meets the target conditions based on the second candidate perception data.

[0029] In some embodiments, the robotic arm is installed in a storage bin on the top of the fuselage; the first control module is also used to control the first acquisition component to collect first perception data in the current environment of the robot when the robotic arm is stored in the storage bin; the second control module is also used to control the robotic arm to perform an out-of-bin operation, and control the robotic arm to drive the second acquisition component to collect information on the target object to obtain second perception data.

[0030] In some embodiments, the first control module is also used to control the robot to perform a cleaning task when the robotic arm is retracted into the receiving bin; and to control the first acquisition component to collect first perception data in the current environment of the robot while the robot is performing the cleaning task.

[0031] An embodiment of the present application provides a robot, including a processor and a memory, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps in the above-mentioned robot grasping control method are implemented.

[0032] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the above-mentioned robot grasping control method are implemented.

[0033] An embodiment of the present application provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor, the steps in the above-mentioned robot grasping control method are implemented.

[0034] In the embodiment of the present application, the first acquisition component is controlled to collect the first sensed data in the current environment where the robot is located, and the mechanical arm is controlled to drive the second acquisition component to collect the second sensed data in the current environment, so as to control the mechanical arm to grasp the target object based on the first sensed data and the second sensed data. In this way, the target object can be grasped by the first sensed data and the second sensed data obtained by the first acquisition component and the second acquisition component at different positions, and the target object can be grasped according to the data from different perspectives, which reduces the possibility of misidentification caused by the limited information obtained at a single perspective, and improves the accuracy and flexibility of the grasping control method.

[0035] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0037] Figure 1 A schematic diagram of the implementation process of a robot grasping control method provided in an embodiment of the present application Figure 1 ;

[0038] Figure 2 A schematic diagram of the implementation process of a robot grasping control method provided in an embodiment of the present application Figure 2 ;

[0039] Figure 3 A schematic diagram of the structure of a robot gripping control device provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of the structure of a robot provided in an embodiment of the present application Figure 1 ;

[0041] Figure 5 A schematic diagram of the structure of a robot provided in an embodiment of the present application Figure 2 ;

[0042] Fig. 6A A schematic diagram of the structure of a robot provided in an embodiment of the present application Figure 3 ;

[0043] Figure 6B A schematic diagram of the structure of a robot provided in an embodiment of the present application Figure 4 . DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0045] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0046] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0047] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art in the field to which the embodiments of the present application belong. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0048] In recent years, robot vision technology has made significant progress in grasping operations. Early grasping systems usually rely on pre-set rules and simple two-dimensional image data, and can only handle objects with fixed positions or simple structures. However, with the rapid development of deep learning and three-dimensional visual perception technology, robots have significantly improved their ability to identify, locate, and accurately grasp irregular objects.

[0049] Modern robot vision systems use depth sensors and high-resolution cameras to obtain the three-dimensional structure information of objects, and combined with advanced visual algorithms, can quickly generate the object's grasping points and optimal grasping strategies. This capability enables robots to accurately identify objects with variable shapes or partial occlusions in complex dynamic scenes, and flexibly adjust the posture of the robotic arm to achieve stable grasping.

[0050] However, current robots usually place sensing devices (e.g., RGB cameras, laser radars) on the body to identify grasped objects, or place sensing devices on the robotic arm to identify grasped objects. When identifying and determining the target object, the judgment angle is single, and the position and posture of the camera and the robotic arm are inconsistent, which can easily lead to the mismatch between the camera's recognition and judgment results of the grasping point and the robotic arm. Even if sensing elements are placed on both the body and the robotic arm, the results of the two are not fused and perceived, and only the target operations such as obstacle avoidance are performed based on the sensing camera on the robotic arm. The above methods have the problems of low accuracy and poor flexibility.

[0051] The embodiment of the present application provides a robot grasping control method, which can be applied to the robot. The robot can be a fixed-position robot or a mobile robot; a mobile robot refers to a robot that can move autonomously. For example, the robot can include but is not limited to at least one of a cleaning robot (such as a sweeper, a scrubber, a mop, a mop-and-washer all-in-one, etc.), a guide robot, a service robot, etc. Figure 1 A schematic diagram of the implementation process of a robot grasping control method provided in an embodiment of the present application Figure 1 , wherein the robot includes a body, a mechanical arm arranged on the top of the body, a first collection component arranged on the body, and a second collection component arranged on the mechanical arm, such as Figure 1 As shown, the method includes the following steps S101 to S103:

[0052] Step S101, controlling the first acquisition component to collect first perception data in the current environment of the robot.

[0053] Here, the first acquisition component is a component having the functions of acquiring data, recording data and transmitting data. The current environment where the robot is located may include but is not limited to one of the following: indoors, outdoors.

[0054] The location of the first collection component on the fuselage is not limited. In some embodiments, the first collection component can be arranged in front of the fuselage, on the side, and / or on the top.

[0055] In some embodiments, the number of the first acquisition component may be at least one. The first acquisition component may include but is not limited to at least one of the following: an RGB camera, an infrared camera, a depth camera, a structured light camera, etc. It should be noted that the first acquisition component may be a combination of one or more of the above cameras, and this application does not limit the first acquisition component.

[0056] In some embodiments, the first perception data is data obtained by perceiving the current environment in which the robot is located. The number of the first perception data may be at least one. The first perception data may include but is not limited to at least one of the following: image data, point cloud data, radar data, etc. The image data may be of any suitable type, for example, an infrared image, a visual image.

[0057] In some embodiments, the fuselage includes a control component, an identification module, a floor cleaning module, and a floor motion module. The control component has a control function and can issue control instructions to the components of the robot to control the robot. The identification module can identify the input data. The floor cleaning module and the floor motion module can clean stains by moving the fuselage. In implementation, the control component may include but is not limited to one of the following: a central processing unit (CPU), a microcontroller (MCU), a single chip microcomputer (SCM), etc. The CPU is the computing and control core of the computer system and the final execution unit for information processing and program operation. The MCU is a chip-level computer that appropriately reduces the frequency and specifications of the CPU and integrates peripheral interfaces such as memory and counters on a single chip, which can be used for different applications. Different combination controls can be performed. SCM is a microcomputer that integrates the main computer functional components such as CPU, random access memory (RAM), read-only memory (ROM), input / output ports, etc. on an integrated circuit chip.

[0058] In some embodiments, the robot arm refers to a complex system with high precision, multiple inputs and multiple outputs, high nonlinearity, and strong coupling. In some embodiments, the robot arm is arranged on the top of the fuselage, and the robot arm may include at least one joint and an end effector. The robot arm may grip a graspable target object through the end effector.

[0059] In some embodiments, the robotic arm is installed in a storage compartment on the top of the fuselage. When the robotic arm is stored in the storage compartment, the first acquisition component can be controlled to collect first perception data in the current environment of the robot.

[0060] Step S102, controlling the robotic arm to drive the second acquisition component to collect second perception data in the current environment.

[0061] Here, the second acquisition component is a component having the functions of acquiring data, recording data, and transmitting data. In some embodiments, the second acquisition component and the first acquisition component may be of the same type or different type.

[0062] In some embodiments, the number of the second acquisition component may be at least one. The second acquisition component may include but is not limited to at least one of the following: an RGB camera, an infrared camera, a depth camera, a structured light camera, etc. It should be noted that the second acquisition component may be a combination of one or more of the above cameras, and this application does not limit the second acquisition component.

[0063] In some embodiments, the second perception data is data obtained by perceiving the current environment in which the robot is located. The number of the second perception data may be at least one. The second perception data may include but is not limited to at least one of the following: image data, point cloud data, radar data, etc. The image data may be of any suitable type, for example, an infrared image, a visual image.

[0064] In some embodiments, the second perception data and the first perception data may be data of the same type or data of different types. The second perception data and the first perception data may be collected from different perspectives.

[0065] Step S103, based on the first perception data and the second perception data, control the robotic arm to grasp the target object that can be grasped in the current environment.

[0066] Here, the robotic arm can be controlled simultaneously to grasp the target object that can be grasped in the current environment based on the first perception data and the second perception data.

[0067] In some embodiments, a first geometric feature corresponding to the target object may be determined based on the first sensing data, and a second geometric feature corresponding to the target object may be determined based on the second sensing data, so as to control the robotic arm to grasp the graspable target object in the current environment based on the first geometric feature and the second geometric feature. The geometric features (including the first geometric feature, the second geometric feature, and the geometric features mentioned below) may include but are not limited to at least one of the following: normal information, curvature information, etc.

[0068] In some embodiments, the grasping point of the robot arm on the target object when grasping the target object can be determined based on the first perception data and the second perception data, and then the joint angle of the robot arm is adjusted through the inverse kinematics algorithm, so as to control the robot arm to grasp the target object that can be grasped in the current environment. Among them, inverse kinematics can be understood as knowing the position and posture of the robot arm and finding the angles of each joint of the robot arm.

[0069] In some embodiments, a strategy generation model can be trained based on the first perception data, the second perception data, and the grasping point at the historical moment, and then the first perception data and the second perception data can be input to the trained strategy generation model, and the strategy generation model can output the grasping point, so as to control the robot arm to grasp the graspable target object in the current environment according to the grasping point. The strategy generation model can be any suitable model, for example, a supervised learning model, a reinforcement learning model, a deep learning model, etc.

[0070] In some embodiments, a first recognition result corresponding to the first perception data and a second recognition result corresponding to the second perception data can be determined, so as to control the robotic arm to grasp a graspable target object in the current environment based on the first recognition result and the second recognition result.

[0071] In the embodiment of the present application, the first acquisition component is controlled to collect the first sense data in the current environment where the robot is located, and the mechanical arm is controlled to drive the second acquisition component to collect the second sense data in the current environment, so as to control the mechanical arm to grasp the target object based on the first sense data and the second sense data. In this way, the target object can be grasped by the first sense data and the second sense data obtained by the first acquisition component and the second acquisition component at different positions, and the target object can be grasped according to the data from different perspectives, which reduces the possibility of misidentification caused by the limited information obtained at a single perspective, and improves the accuracy and flexibility of the grasping control method.

[0072] In some embodiments, the above step S102 may include the following steps S121 and S122:

[0073] Step S121, based on the first perception data, identify whether there is a graspable target object in the current environment to obtain a recognition result.

[0074] Here, the target object refers to an object that can be grasped by the robot arm. The target object can be of any suitable type, such as a shoe, a cup, a toy, etc.

[0075] In some embodiments, the recognition result may indicate whether there is a graspable target object in the current environment. The recognition result may indicate whether there is a graspable target object in the current environment, or may indicate whether there is no graspable target object in the current environment.

[0076] In some embodiments, the object identified in the first perception data may be of any suitable type, such as a pet, a coffee table, a cup, etc. It should be noted that the object identified in the first perception data and the graspable target object may be the same object or different objects.

[0077] In some embodiments, when identifying whether there is a graspable target object in the current environment, the object grasping capability of the robot arm needs to be considered. The object grasping capability of the robot arm can be used to characterize the attribute information corresponding to the object that the robot arm can grasp. The attribute information may include but is not limited to at least one of the following: weight, volume, aspect ratio, surface flatness, etc.

[0078] In some embodiments, the grasping ability of the robot arm is related to the structural parameters of the robot arm. The structural parameters of the robot arm may include but are not limited to at least one of the following: the joint type of the robot arm, the connecting rod length of the robot arm, the joint angle limit of the robot arm, etc.

[0079] In some implementations, the first perception data may be preprocessed to obtain preprocessed first perception data, and then the presence of a graspable target object in the current environment may be identified based on the preprocessed first perception data to obtain a recognition result. The preprocessing may include but is not limited to at least one of the following: denoising, enhancement, cropping, filtering, etc.

[0080] In some implementations, the first perception data may be processed by a deep learning framework or a point cloud processing library to identify whether there is a graspable target object in the current environment to obtain a recognition result. The deep learning framework may include but is not limited to: PyTorch, TensorFlow, etc. The point cloud processing library may include but is not limited to: Open3D, PCL, etc.

[0081] In some embodiments, the recognition module of the fuselage has the ability to analyze data, and the recognition of data can be realized through a recognition algorithm. The recognition module is called by the control component, and the recognition module identifies whether there is a graspable target object in the current environment based on the first perception data to obtain a recognition result. The recognition algorithm can be an image, point cloud or other recognition algorithm based on deep learning, such as the YOLO algorithm, the Bevformer algorithm, or an image, point cloud algorithm based on manually extracted features, such as the Scale-invariant feature transform (SIFT) algorithm, the Iterative Closest Point (ICP) algorithm, etc. This application does not limit the recognition algorithm used by the recognition module.

[0082] In some embodiments, a first correspondence between the perception data and the recognition result can be established, so that after determining the first perception data, it is possible to identify whether there is a graspable target object in the current environment based on the first correspondence and the first perception data to obtain a recognition result.

[0083] Step S122, when the recognition result indicates that there is a target object, control the robotic arm to drive the second acquisition component to collect information on the target object to obtain second perception data.

[0084] Here, it should be noted that the sweeping robot can control the first acquisition component to collect first perception data in the current environment of the robot during the cleaning task, so as to determine the recognition result according to the first perception data, and when the recognition result indicates the existence of the target object, control the mechanical arm to perform the exit operation from the storage bin, and control the mechanical arm to drive the second acquisition component to collect information on the target object to obtain the second perception data. The mechanical arm can be placed in the storage bin without detecting the target object, reducing the possibility of damage to the mechanical arm due to collision.

[0085] In some embodiments, when the recognition result indicates that there is no target object, the first acquisition component can be controlled again to collect first perception data in the current environment of the robot, so as to identify whether there is a graspable target object in the current environment based on the first perception data, and obtain the recognition result again. When the recognition result indicates that there is a target object, the robotic arm is controlled to drive the second acquisition component to collect information on the target object to obtain second perception data.

[0086] In some embodiments, it is possible to identify whether the target object meets the target condition based on the first candidate perception data currently collected by the second acquisition component, and if the target object meets the target condition, the first candidate perception data is used as the second perception data. The target condition includes at least one of the following: the proportion of the target perception data corresponding to the target object in the first candidate perception data is greater than or equal to a preset ratio, and the target object is located within the visual cone corresponding to the second acquisition component.

[0087] In some embodiments, the robotic arm is installed in a storage bin on top of the fuselage, and the robotic arm can be controlled to perform an exit operation, and the robotic arm can be controlled to drive the second acquisition component to collect information on the target object to obtain second perception data.

[0088] In the above embodiment, when the recognition result obtained by recognizing the current environment according to the first perception data indicates that there is a target object, the control arm drives the second acquisition component to collect information about the target object to obtain the second perception data. In this way, the collection of the second perception data can be triggered only when it is determined that there is a target object, which reduces the computing resources consumed by executing the grasping control method and improves the flexibility of the grasping control method.

[0089] In some embodiments, the above step S103 may include the following steps S131 and S132:

[0090] Step S131, determining a first recognition result corresponding to the first perception data and a second recognition result corresponding to the second perception data.

[0091] Here, the recognition result (including the first recognition result, the second recognition result and the recognition results mentioned later) refers to the information obtained by recognizing the target object.

[0092] In some embodiments, the recognition result may include but is not limited to at least one of the following: the position of the target object, the posture of the target object, the category corresponding to the target object, the target grasping point corresponding to the target object, etc. The category corresponding to the target object may be any suitable category, for example, slippers, bowls, building blocks, etc.

[0093] In some embodiments, a three-dimensional coordinate system can be established to represent the position of the target object and the target grasping point corresponding to the target object in the form of three-dimensional coordinates. For example, the position of the target object can be (8, 3, 5), and the target grasping point corresponding to the target object can be (3.5, 6, 4).

[0094] In some embodiments, when the target object is a regular geometric body, the position of the target object can be determined according to the center of the target object. When the target object is an irregular geometric body, the target object can be simplified into a regular geometric body according to its shape, and then the position of the target object can be determined according to the center of the regular geometric body corresponding to the target object.

[0095] In some embodiments, the recognition module may be called by the control component, and the first perception data and the second perception data may be input into the recognition module, and the first recognition result corresponding to the first perception data and the second recognition result corresponding to the second perception data may be determined by the recognition algorithm in the recognition module. The recognition algorithm may include but is not limited to at least one of the following: YOLO algorithm, Bevformer algorithm, SIFT algorithm, ICP algorithm, etc.

[0096] Step S132: Based on the first recognition result and the second recognition result, control the robot arm to grasp the target object.

[0097] Here, the robot arm may be simultaneously controlled to grasp the target object based on the first recognition result and the second recognition result.

[0098] In some embodiments, the strategy generation rules can be learned from historical data, and then the strategy generation model is trained using the rules. Then, the first recognition result and the second recognition result are input to the strategy generation model, and the output of the model is the grasping point, so that the robot arm can be controlled to grasp the target object according to the grasping point. The strategy generation model can be any suitable model, such as a supervised learning model, a reinforcement learning model, a deep learning model, etc.

[0099] In some embodiments, the grasping point of the robotic arm on the target object when grasping the target object can be determined based on the first recognition result and the second recognition result, and then the joint angle of the robotic arm can be adjusted through an inverse kinematics algorithm to control the robotic arm to grasp the target object.

[0100] In some embodiments, the first recognition result includes a first target category corresponding to the target object, and the second recognition result includes a second target category corresponding to the target object. When the first target category and the second target category are the same, the robot arm can be controlled to grasp the target object based on the first target grasping point corresponding to the target object. When the first target category and the second target category are different, a first target strategy to be executed can be determined to control the robot arm to grasp the target object based on the first target strategy. The first target grasping point is identified from the first perception data and / or the second perception data based on the first target category.

[0101] In the above embodiment, the robotic arm is controlled to grasp the target object through the first recognition result corresponding to the first perception data and the second recognition result corresponding to the second perception data, and the recognition results corresponding to the perception data of different acquisition perspectives are used to perform grasping control, thereby improving the accuracy of the robot's grasping control.

[0102] In some embodiments, the first recognition result includes a first target category corresponding to the target object, and the second recognition result includes a second target category corresponding to the target object; the above step S132 may include the following step S1321 or step S1322:

[0103] Step S1321, when the first target category and the second target category are the same, control the robotic arm to grasp the target object based on the first target grasping point corresponding to the target object; the first target grasping point is identified from the first perception data and / or the second perception data based on the first target category.

[0104] Here, the first target category may be any suitable category, such as slippers, bowls, etc. The second target category may be any suitable category, such as cups, toys, etc. The first target grasping point is a grasping point identified from the first perception data and / or the second perception data based on the first target category.

[0105] In some implementations, a three-dimensional coordinate system may be established to represent the first target grasping point in the form of three-dimensional coordinates. For example, the first target grasping point may be (5, 6.6, 4).

[0106] In some embodiments, when controlling a robotic arm to grasp a target object based on a first target grasping point corresponding to the target object, the position of the target object can be converted from a camera coordinate system to a base coordinate system of the robotic arm, and then the coordinates of the first target grasping point are converted into a posture that the end effector of the robotic arm needs to reach. Then, the motion path of the robotic arm is planned through an inverse kinematics algorithm, so that the robotic arm reaches the first target grasping point based on the planned motion path. After the robotic arm reaches the first target grasping point, the end effector is controlled to perform a grasping action to grasp the target object.

[0107] In some embodiments, when a first recognition result is determined based on first perception data, the first recognition result may include a first target grasping point, and when a second recognition result is determined based on second perception data, the second recognition result may include the first target grasping point. Therefore, the first target grasping point can be identified from the first perception data and / or the second perception data.

[0108] In some embodiments, the first grasping point can be determined based on the first perception data, and the second grasping point can be determined based on the second perception data, and then the first target grasping point can be identified based on the first grasping point and the second grasping point. In implementation, when the position of the first grasping point and the position of the second grasping point are the same, the first grasping point or the second grasping point can be used as the first target grasping point. When the position of the first grasping point and the position of the second grasping point are different, the first grasping point and the second grasping point can be connected by a line and the midpoint of the line can be calculated, and then the midpoint can be used as the first target grasping point.

[0109] Step S1322, when the first target category and the second target category are different, determine a first target strategy to be executed, so as to control the robot arm to grasp the target object based on the first target strategy.

[0110] Here, the first target strategy is a strategy for grasping the target object. The first target strategy to be executed will be determined only when the first target category and the second target category are different.

[0111] Exemplarily, the first target category is slippers, and the second target category is sports shoes. The first target category and the second target category are different, and a first target strategy to be executed needs to be determined to control the robotic arm to grasp the target object based on the first target strategy.

[0112] The method for determining the first target strategy may include, but is not limited to: determining the first target strategy through machine learning, determining the first target strategy through multi-objective optimization, etc. For example, the strategy generation rules can be learned from historical data, and then the strategy generation model can be trained using the rules. Then, the first target category and / or the second target category are input into the strategy generation model, and the output of the model is the first target strategy. Among them, the strategy generation model can be any suitable model, such as a supervised learning model, a reinforcement learning model, a deep learning model, etc. For another example, the strategy determination problem can be modeled as a multi-objective optimization problem based on the first target category and / or the second target category, and then the optimization algorithm can be used to solve the multi-objective optimization problem to determine the target obstacle avoidance strategy. Among them, the goal can be to grasp the target object. The optimization algorithm may include, but is not limited to: genetic algorithm, particle swarm optimization algorithm, etc.

[0113] In some embodiments, controlling the robotic arm to grasp the target object based on the first target strategy may include one of the following: controlling the robotic arm to give up grasping the target object; controlling the robotic arm to grasp the target object based on the first target grasping point; controlling the robotic arm to grasp the target object based on the second target grasping point; controlling the robotic arm and / or the fuselage to detour from the current first position to the second position of the target object, controlling the first acquisition component to collect information on the target object at the second position to obtain third perception data, and controlling the robotic arm to drive the second acquisition component to collect information on the target object to obtain fourth perception data, so as to control the robotic arm to grasp the target object based on the third perception data and the fourth perception data. The first target grasping point is identified from the first perception data and / or the second perception data based on the first target category. The second target grasping point is identified from the first perception data and / or the second perception data based on the second target category.

[0114] In the above embodiments, when the first target category and the second target category are the same or different, different processing flows can be used for grasping control according to the actual usage scenario of the robot, thereby improving the flexibility of the grasping control method.

[0115] In some embodiments, the “controlling the robotic arm to grasp the target object based on the first target strategy” in the above step S1322 may include one of the following steps S1322a to S1322d:

[0116] Step S1322a, controlling the robotic arm to give up grasping the target object.

[0117] Here, when the first target category and the second target category are different, it indicates that the robot may make an error in its grasping judgment of the target object, and therefore, the robot arm may be controlled to give up grasping the target object.

[0118] In some embodiments, the control component can be used to control the robotic arm to give up grasping the target object.

[0119] In some embodiments, the robotic arm is installed in a storage bin on top of the fuselage, and after the robotic arm is controlled to give up grabbing the target object, the robotic arm can be further controlled to be retracted into the storage bin.

[0120] Step S1322b, controlling the robotic arm to grasp the target object based on the first target grasping point; the first target grasping point is identified from the first perception data and / or the second perception data based on the first target category.

[0121] Here, the first target grasping point is a grasping point identified from the first perception data and / or the second perception data based on the first target category.

[0122] It should be noted that the first acquisition component can be arranged in front of the fuselage, and the first perception data of the first acquisition component is more accurate in determining the grasping point of the target object with more details in the side view. For example, for a target object with a target category of a toy duck, the first target grasping point obtained based on the first target category recognition will more accurately grasp the target object.

[0123] In some implementations, a three-dimensional coordinate system may be established to represent the first target grasping point in the form of three-dimensional coordinates. For example, the first target grasping point may be (5, 6.6, 4).

[0124] In some embodiments, when controlling a robotic arm to grasp a target object based on a first target grasping point corresponding to the target object, the position of the target object can be converted from a camera coordinate system to a base coordinate system of the robotic arm, and then the coordinates of the first target grasping point are converted into a posture that the end effector of the robotic arm needs to reach. Then, the motion path of the robotic arm is planned through an inverse kinematics algorithm, so that the robotic arm reaches the first target grasping point based on the planned motion path. After the robotic arm reaches the first target grasping point, the end effector is controlled to perform a grasping action to grasp the target object.

[0125] Step S1322c, controlling the robotic arm to grasp the target object based on the second target grasping point; the second target grasping point is identified from the first perception data and / or the second perception data based on the second target category.

[0126] Here, the second target grasping point is a grasping point identified from the first perception data and / or the second perception data based on the second target category.

[0127] It should be noted that the second acquisition component is provided on the robot arm, and the second perception data of the second acquisition component is more accurate in determining the grasping point of the target object with more details in the top-down perspective. For example, for a target object whose target category is a pet bowl, the second target grasping point obtained based on the second target category recognition will more accurately grasp the target object.

[0128] In some implementations, a three-dimensional coordinate system may be established to represent the second target grasping point in the form of three-dimensional coordinates. For example, the second target grasping point may be (3, 3.6, 5).

[0129] In some embodiments, when controlling a robotic arm to grasp a target object based on a second target grasping point corresponding to the target object, the position of the target object can be converted from a camera coordinate system to a base coordinate system of the robotic arm, and then the coordinates of the second target grasping point are converted into the posture that the end effector of the robotic arm needs to reach. Then, the motion path of the robotic arm is planned through an inverse kinematics algorithm, so that the robotic arm reaches the second target grasping point based on the planned motion path. After the robotic arm reaches the second target grasping point, the end effector is controlled to perform a grasping action to grasp the target object.

[0130] In some embodiments, when a first recognition result is determined based on first perception data, the first recognition result may include a first target grasping point, and when a second recognition result is determined based on second perception data, the second recognition result may include a second target grasping point. Therefore, the second target grasping point can be identified from the first perception data and / or the second perception data.

[0131] Step S1322d, control the robotic arm and / or the body to move from the current first position to the second position of the target object, control the first acquisition component at the second position to collect information on the target object to obtain third perception data, and control the robotic arm to drive the second acquisition component to collect information on the target object to obtain fourth perception data, so as to control the robotic arm to grasp the target object based on the third perception data and the fourth perception data.

[0132] Here, the first position refers to the current position of the robot's mechanical arm and / or body. The second position refers to the position of the robot's mechanical arm and / or body after circling the target object.

[0133] It should be noted that the shooting angle of the first perception data and the shooting angle of the second perception data will affect the recognition of the first target category and the second target category. When the first target category and the second target category are different, the first acquisition component and the second acquisition component can be controlled to change positions to obtain perception data respectively for re-identification.

[0134] In some embodiments, the shooting angle at the first position is different from the shooting angle at the second position, and the perception data acquired by the robotic arm and / or the body at the first position is different from the perception data acquired by the robotic arm and / or the body at the second position.

[0135] In some embodiments, the fuselage and / or the robotic arm may be simplified into a regular geometric body according to the shape of the fuselage and / or the robotic arm, and then the first position may be determined according to the center of the regular geometric body corresponding to the fuselage and / or the robotic arm.

[0136] In some embodiments, the third sensory data is data obtained by the first acquisition component from acquiring information about the target object. The number of the third sensory data may be at least one. The third sensory data may include, but is not limited to, at least one of the following: image data, point cloud data, radar data, etc. The image data may be of any suitable type, for example, an infrared image, a visual image.

[0137] In some embodiments, the fourth sensory data is data obtained by the second acquisition component collecting information about the target object. The number of the fourth sensory data may be at least one. The fourth sensory data may include but is not limited to at least one of the following: image data, point cloud data, radar data, etc. The image data may be of any suitable type, for example, an infrared image, a visual image.

[0138] In some embodiments, the first acquisition component and / or the second acquisition component can adjust the position of the first acquisition component and / or the second acquisition component by telescoping or rotating. While controlling the robotic arm and / or the fuselage to move from the current first position to the second position of the target object, the first acquisition component and / or the second acquisition component can be controlled to adjust the position, so that the third perception data and the fourth perception data with different shooting angles can be obtained with less change in the position of the robotic arm and / or the fuselage.

[0139] In some embodiments, a third recognition result corresponding to the third perception data and a fourth recognition result corresponding to the fourth perception data can be determined to control the robotic arm to grasp the target object based on the third recognition result and the fourth recognition result.

[0140] In some embodiments, when the third recognition result corresponding to the third perception data and the fourth recognition result corresponding to the fourth perception data are the same, the robot arm can be controlled to grasp the target object based on the third target grasping point or the fourth target grasping point. The third target grasping point is obtained by recognizing the third perception data and / or the fourth perception data based on the third target category in the third recognition result, and the fourth target grasping point is obtained by recognizing the third perception data and / or the fourth perception data based on the fourth target category in the fourth recognition result.

[0141] In some embodiments, when the third recognition result and the fourth recognition result are different, and the number of times the first acquisition component or the second acquisition component collects information on the target object exceeds a threshold number, the robotic arm can be controlled to grasp the target object based on the second target strategy. The second target strategy includes one of the following: controlling the robotic arm to give up grasping the target object; controlling the robotic arm to grasp the target object based on the third target grasping point; and controlling the robotic arm to grasp the target object based on the fourth target grasping point.

[0142] In the above embodiment, when the first target category and the second target category are different, the robotic arm is controlled to grasp the target object through the first target strategy including multiple strategies. The required first target strategy can be selected according to the actual usage scenario, thereby improving the compatibility of the grasping control method.

[0143] In some embodiments, the above step S1322d of “controlling the robotic arm to grasp the target object based on the third sensing data and the fourth sensing data” may include the following step S1322d1 or step S1322d2:

[0144] Step S1322d1, when the third recognition result corresponding to the third perception data and the fourth recognition result corresponding to the fourth perception data are the same, control the robotic arm to grasp the target object based on the third target grasping point or the fourth target grasping point; the third target grasping point is identified from the third perception data and / or the fourth perception data based on the third target category in the third recognition result, and the fourth target grasping point is identified from the third perception data and / or the fourth perception data based on the fourth target category in the fourth recognition result.

[0145] Here, the recognition result (including the third recognition result, the fourth recognition result and the recognition results mentioned later) may include but is not limited to at least one of the following: the position of the target object, the posture of the target object, the category corresponding to the target object, the target grasping point corresponding to the target object, etc. The category corresponding to the target object may be any appropriate category, for example, slippers, bowls, building blocks, etc.

[0146] In some embodiments, the third recognition result includes a third target category corresponding to the target object, the fourth recognition result includes a fourth target category corresponding to the target object, the third recognition result corresponding to the third perception data and the fourth recognition result corresponding to the fourth perception data are the same, and the third target category and the fourth target category may be the same.

[0147] In some embodiments, the third target grasping point is a grasping point identified from the third perception data and / or the fourth perception data based on the third target category in the third recognition result. The fourth target grasping point is a grasping point identified from the third perception data and / or the fourth perception data based on the fourth target category in the fourth recognition result.

[0148] In some embodiments, when controlling the robotic arm to grasp the target object based on the third target grasping point or the fourth target grasping point, the position of the target object can be converted from the camera coordinate system to the base coordinate system of the robotic arm, and then the coordinates of the third target grasping point or the fourth target grasping point are converted into the posture that the end effector of the robotic arm needs to reach, and then the motion path of the robotic arm is planned through the inverse kinematics algorithm, so that the robotic arm reaches the third target grasping point or the fourth target grasping point based on the planned motion path. After the robotic arm reaches the third target grasping point or the fourth target grasping point, the end effector is controlled to perform a grasping action to grasp the target object.

[0149] Step S1322d2, when the third recognition result is different from the fourth recognition result and the number of times the first acquisition component or the second acquisition component collects information on the target object exceeds the number threshold, the robotic arm is controlled to grasp the target object based on the second target strategy, and the second target strategy includes one of the following: controlling the robotic arm to give up grasping the target object; controlling the robotic arm to grasp the target object based on the third target grasping point; controlling the robotic arm to grasp the target object based on the fourth target grasping point.

[0150] Here, the number of times the second collection component collects information from the target object can be any appropriate value, for example, 3, 2, etc. The number threshold can be any appropriate value, for example, 5, 6, etc.

[0151] Exemplarily, the number of times the second acquisition component collects information on the target object is 6, and the number threshold is 5, which satisfies that the number of times the second acquisition component collects information on the target object exceeds the number threshold.

[0152] In some embodiments, the second target strategy includes one of the following: controlling the robotic arm to give up grasping the target object; controlling the robotic arm to grasp the target object based on the third target grasping point; and controlling the robotic arm to grasp the target object based on the fourth target grasping point.

[0153] In some embodiments, the mechanical arm is installed in the storage bin on the top of the fuselage, and the control component can be used to control the mechanical arm to give up grabbing the target object. After the mechanical arm is controlled to give up grabbing the target object, the mechanical arm can be further controlled to be stored in the storage bin.

[0154] In some embodiments, when controlling the robotic arm to grasp the target object based on the third target grasping point or the fourth target grasping point, the position of the target object can be converted from the camera coordinate system to the base coordinate system of the robotic arm, and then the coordinates of the third target grasping point or the fourth target grasping point are converted into the posture that the end effector of the robotic arm needs to reach, and then the motion path of the robotic arm is planned through the inverse kinematics algorithm, so that the robotic arm reaches the third target grasping point or the fourth target grasping point based on the planned motion path. After the robotic arm reaches the third target grasping point or the fourth target grasping point, the end effector is controlled to perform a grasping action to grasp the target object.

[0155] In some embodiments, when the third recognition result and the fourth recognition result are different and the number of times the first acquisition component or the second acquisition component collects information on the target object does not exceed a threshold number, the robotic arm and / or the fuselage can be controlled to detour from the current second position to the third position of the target object, and the first acquisition component can be controlled at the third position to collect information on the target object to obtain fifth perception data, and the robotic arm can be controlled to drive the second acquisition component to collect information on the target object to obtain sixth perception data, so as to control the robotic arm to grasp the target object based on the fifth perception data and the sixth perception data.

[0156] In the above embodiment, when the third recognition result and the fourth recognition result are the same or different, grasping control is performed through different processing flows. When multiple identifications are performed for grasping control, by setting a number threshold, the possibility of grasping failure due to recognition errors can be reduced while the final grasping strategy can be quickly determined, thereby improving the response speed of the robot.

[0157] In some embodiments, the above step S122 of “controlling the robotic arm to drive the second acquisition component to collect information from the target object to obtain the second perception data” may include the following steps S1221 and S1222:

[0158] Step S1221, based on the first candidate perception data currently collected by the second acquisition component, identify whether the target object meets the target condition; the target condition includes at least one of the following: the proportion of the target perception data corresponding to the target object in the first candidate perception data is greater than or equal to a preset ratio, and the target object is located in the visual cone corresponding to the second acquisition component.

[0159] Here, the first candidate perception data is the data currently collected by the second acquisition component. The number of the first candidate perception data can be at least one. The first candidate perception data can include but is not limited to at least one of the following: image data, point cloud data, radar data, etc. The image data can be of any suitable type, for example, infrared images, visual images.

[0160] In some implementations, the first candidate perception data and the second perception data may be the same data or different data.

[0161] In some embodiments, the target perception data is perception data used to characterize the target object. The proportion of the target perception data in the first candidate perception data can be any suitable size, for example, 30%, 0.6, etc. The preset ratio can be any suitable size, for example, 75%, 0.8, etc.

[0162] In some implementations, the target object may be selected in the first candidate perception data by a marking box, and then the proportion of the target perception data selected by the marking box in the first candidate perception data is calculated to identify whether the target object meets the target condition.

[0163] In some embodiments, whether the target object meets the target condition can be identified based on the position of the annotation box in the first candidate perception data. When the annotation box is located in the middle area of ​​the first candidate perception data, it can be determined that the proportion of the target perception data corresponding to the target object in the first candidate perception data is greater than or equal to a preset ratio.

[0164] In some embodiments, the second acquisition component has a corresponding viewing cone. The viewing cone refers to the area that the second acquisition component can see in three-dimensional space. The target condition may include that the target object is located within the viewing cone corresponding to the second acquisition component.

[0165] In some embodiments, the view cone is a three-dimensional space region defined by the camera's viewing angle, aspect ratio, near clipping plane, and far clipping plane. The positional relationship between the target object and the view cone corresponding to the second acquisition component can be determined by a view cone culling algorithm. The view cone culling algorithm can determine whether the bounding box of the target object intersects with the six clipping planes of the view cone. If the target object is completely outside the view cone, it is culled. If the object intersects with the view cone or is inside the view cone, the target object is retained. After using the view cone culling algorithm, if the target object is not culled, the target object is located within the view cone corresponding to the second acquisition component.

[0166] Step S1222: When the target object meets the target condition, the first candidate perception data is used as the second perception data.

[0167] Here, the target object satisfies the target condition including at least one of the following: the proportion of target perception data corresponding to the target object in the first candidate perception data is greater than or equal to a preset ratio, and the target object is located within the visual cone corresponding to the second acquisition component.

[0168] Exemplarily, the target perception data accounts for 80% of the first candidate perception data. The preset ratio is 75%. If the target perception data corresponding to the target object accounts for more than or equal to the preset ratio in the first candidate perception data, the target object meets the target condition.

[0169] Exemplarily, after using the frustum culling algorithm, if the target object is not culled, the target object is located within the frustum corresponding to the second acquisition component. At this time, the target object meets the target condition.

[0170] In some implementations, the first candidate perception data may be preprocessed to obtain preprocessed first candidate perception data, and then the preprocessed first candidate perception data may be used as the second perception data. The preprocessing may include but is not limited to at least one of the following: denoising, enhancement, cropping, filtering, etc.

[0171] In the above embodiment, by identifying whether the target object in the first candidate perception data currently collected by the second acquisition component meets the target condition, and using the first candidate perception data that meets the target condition as the second perception data, the accuracy of the second perception data is improved, thereby improving the accuracy of the grasping control.

[0172] In some embodiments, the above method further includes the following step S1223:

[0173] Step S1223, when the target object does not meet the target condition, control the robotic arm and / or the fuselage to drive the second acquisition component to move and collect information on the target object to obtain second candidate perception data, so as to identify whether the target object meets the target condition based on the second candidate perception data.

[0174] Here, the target object does not meet the target condition including but not limited to one of the following: the proportion of target perception data corresponding to the target object in the first candidate perception data is less than a preset ratio, and the target object is not located in the visual cone corresponding to the second acquisition component.

[0175] Exemplarily, the target perception data accounts for 50% of the first candidate perception data. The preset ratio is 80%. If the target perception data corresponding to the target object accounts for less than the preset ratio in the first candidate perception data, the target object does not meet the target condition.

[0176] Exemplarily, after using the frustum culling algorithm, the target object is culled, and the target object is not located within the frustum corresponding to the second acquisition component. At this time, the target object does not meet the target condition.

[0177] In some embodiments, the position of the second acquisition component can be adjusted by telescoping or rotating. When the target object does not meet the target conditions, the robotic arm and / or the fuselage can be controlled to detour from the current first position to the second position of the target object. At the same time, the second acquisition component is controlled to adjust its position, so that the robotic arm and / or the fuselage can be controlled to drive the second acquisition component to move and collect information on the target object to obtain second candidate perception data.

[0178] In the above embodiment, when the target object does not meet the target condition, the second acquisition component is driven to move to obtain second candidate perception data, so as to identify whether the target object meets the target condition based on the second candidate perception data. The second perception data that meets the target condition can be continuously acquired, thereby improving the accuracy of collecting the second perception data.

[0179] In some embodiments, the mechanical arm is installed in a storage compartment on top of the fuselage; the above step S101 may include the following step S111, and the "controlling the mechanical arm to drive the second acquisition component to collect information from the target object to obtain the second perception data" in the above step S122 may include the following step S1224:

[0180] Step S111, when the robotic arm is put into the storage bin, control the first acquisition component to collect first perception data in the current environment where the robot is located.

[0181] Here, the robotic arm is installed in a storage bin on the top of the fuselage. When collecting the first perception data, the robotic arm is retracted into the storage bin and the operation of grabbing the target object is not performed.

[0182] In some embodiments, when the robotic arm is retracted into the storage bin, the control component can be used to control the first acquisition component to collect first perception data in the current environment of the robot.

[0183] In some embodiments, when the robotic arm is retracted into the storage bin, the robot is controlled to perform a cleaning task, so that during the robot's performance of the cleaning task, the first acquisition component is controlled to collect first perception data in the current environment of the robot.

[0184] Step S1224, control the robotic arm to perform the out-of-warehouse operation, and control the robotic arm to drive the second acquisition component to collect information on the target object to obtain second perception data.

[0185] Here, when the robotic arm is stored in the storage bin, it is impossible to collect information on the target object. Therefore, the robotic arm can be controlled to perform an exit operation, and the robotic arm can be controlled to drive the second collection component to collect information on the target object to obtain second perception data.

[0186] In some embodiments, the out-of-bin operation of the robot arm refers to an operation of extending the robot arm from the receiving bin. When the recognition result indicates that the target object exists, the robot arm can be controlled to perform the out-of-bin operation.

[0187] In some embodiments, the control component can send control instructions to the robotic arm to control the robotic arm to perform an exit operation, and control the robotic arm to drive the second acquisition component to collect information on the target object to obtain second perception data.

[0188] In some embodiments, the second acquisition component has corresponding camera parameters, and can control the mechanical arm to perform the out-of-bin operation and move to the vicinity of the target object according to the first recognition result and the camera parameters corresponding to the first perception data, so as to control the mechanical arm to drive the second acquisition component to collect information on the target object and obtain the second perception data. The camera parameters may include but are not limited to at least one of the following: focal length, principal point, rotation matrix, translation vector, etc.

[0189] In the above embodiment, on the one hand, when the robotic arm is retracted into the accommodating bin, the first acquisition component is controlled to collect the first perception data in the current environment of the robot, thereby reducing the influence of the occlusion of the robotic arm on the collection of the first perception data and improving the accuracy of the collection of the first perception data; on the other hand, the robotic arm is controlled to perform an out-of-bin operation to collect information to obtain the second perception data, thereby improving the accuracy of the collection of the second perception data.

[0190] In some embodiments, the above step S111 may include the following steps S1111 and S1112:

[0191] Step S1111, when the robotic arm is put into the storage bin, control the robot to perform the cleaning task.

[0192] Here, during the robot's execution of the cleaning task, if the robotic arm is not retracted into the receiving bin, it may block the first acquisition component from collecting the first sensing data or cause collision damage. Therefore, the robot can be controlled to perform the cleaning task when the robotic arm is retracted into the receiving bin.

[0193] In some embodiments, the warehouse entry operation of the robot arm refers to the operation of the robot arm entering the storage bin. The control component can send a control instruction to the robot arm to control the robot arm to perform the warehouse entry operation and store it in the storage bin.

[0194] Step S1112, when the robot performs a cleaning task, control the first acquisition component to collect first perception data in the current environment of the robot.

[0195] Here, the robot can perform cleaning tasks through the ground cleaning module and the ground motion module.

[0196] In some embodiments, when the robot is performing a cleaning task, a control instruction may be sent to the first acquisition component through the control component to control the first acquisition component to collect the first perception data in the current environment of the robot.

[0197] In some embodiments, during the robot's cleaning task, the first acquisition component may be continuously controlled to collect first sensed data in the robot's current environment, or the first acquisition component may be controlled to collect first sensed data in the robot's current environment at intervals of a preset time. The preset time may be any appropriate length, for example, 1 second, 3 seconds, etc.

[0198] In the above embodiment, when the robotic arm is retracted into the accommodating bin, the robot is controlled to perform a cleaning task to collect first perception data, so that the cleaning robot can retract the robotic arm while performing the cleaning task, thereby reducing the possibility of the robotic arm being damaged by collision and improving the safety of the robot.

[0199] The following describes the application of the robot grasping control method provided in the embodiments of the present application in actual scenarios, taking a sweeping robot with a mechanical arm as an example.

[0200] In recent years, robot vision technology has made significant progress in grasping operations. Early grasping systems usually rely on pre-set rules and simple two-dimensional image data, and can only handle objects with fixed positions or simple structures. However, with the rapid development of deep learning and three-dimensional visual perception technology, robots have significantly improved their ability to identify, locate, and accurately grasp irregular objects.

[0201] Modern robot vision systems use depth sensors and high-resolution cameras to obtain the three-dimensional structure information of objects, and combined with advanced visual algorithms, can quickly generate the object's grasping points and optimal grasping strategies. This capability enables robots to accurately identify objects with variable shapes or partial occlusions in complex dynamic scenes, and flexibly adjust the posture of the robotic arm to achieve stable grasping.

[0202] However, current robots usually place sensing devices (e.g., RGB cameras, LiDAR) on the body to identify grasped objects, or place sensing devices on the robotic arm to identify grasped objects. When identifying and determining the target object, the judgment angle is single, and the position and posture of the camera and the robotic arm are inconsistent, which can easily lead to the mismatch between the camera's recognition and judgment results of the grasping point and the robotic arm. Even if sensing elements are placed on both the body and the robotic arm, the results of the two are not fused and perceived, and only the target operations such as obstacle avoidance are performed based on the sensing camera on the robotic arm. The above methods have the problems of low accuracy and poor flexibility.

[0203] Figure 2A schematic diagram of the implementation process of a robot grasping control method provided in an embodiment of the present application Figure 2 ,like Figure 2 As shown, the method may include the following steps S201 to S213:

[0204] Step S201, when the robot arm is stored in the storage bin, controlling the robot to perform a cleaning task;

[0205] Step S202, when the robot performs a cleaning task, controlling the first acquisition component to collect first perception data in the current environment where the robot is located;

[0206] Step S203, identifying whether there is a graspable target object in the current environment according to the first perception data;

[0207] Here, if yes, go to step S204, if no, go to step S202.

[0208] Step S204, controlling the robotic arm to perform an exit operation, and controlling the robotic arm to drive the second acquisition component to collect information on the target object to obtain first candidate perception data;

[0209] Step S205, identifying whether the target object meets the target condition based on the first candidate perception data;

[0210] Here, if yes, go to step S206, if no, go to step S207.

[0211] Step S206, using the first candidate perception data as the second perception data;

[0212] Step S207, controlling the robotic arm and / or the fuselage to drive the second acquisition component to move and collect information from the target object to obtain second candidate perception data;

[0213] Step S208, identifying whether the target object meets the target condition based on the second candidate perception data;

[0214] Here, if yes, go to step S209, if no, go to step S207.

[0215] Step S209, using the second candidate perception data as the second perception data;

[0216] Step S210, determining a first recognition result corresponding to the first perception data and a second recognition result corresponding to the second perception data;

[0217] Step S211, determining whether the first recognition result and the second recognition result are the same;

[0218] Here, if yes, go to step S212, if no, go to step S213.

[0219] Step S212, controlling the robotic arm to grasp the target object based on the first target grasping point corresponding to the target object;

[0220] Step S213, determining a first target strategy to be executed, so as to control the robot arm to grasp the target object based on the first target strategy.

[0221] In the embodiment of the present application, by setting up collection components on the body and the robotic arm to collect the perception data required for grasping control, it is possible to avoid erroneous recognition caused by limited information obtained by a single sensor at a single viewing angle or a single range of viewing angle. At the same time, the robot can retract the robotic arm during the performance of the cleaning task, and when the robotic arm is extended, the perception data from different viewing angles are used to calculate the grasping point, which is more conducive to the execution of grasping control.

[0222] Based on the above embodiments, the present application also provides a robot grasping control device, Figure 3 A schematic diagram of the structure of a robot gripping control device provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the robot grasping control device 300 is applied to the robot, the robot includes a fuselage, a mechanical arm arranged on the top of the fuselage, a first acquisition component arranged on the fuselage, and a second acquisition component arranged on the mechanical arm. The robot grasping control device 300 includes:

[0223] A first control module 301 is used to control a first acquisition component to collect first perception data in the current environment where the robot is located;

[0224] The second control module 302 is used to control the robot arm to drive the second acquisition component to collect second perception data in the current environment;

[0225] The third control module 303 is used to control the robot arm to grasp a graspable target object in the current environment based on the first perception data and the second perception data.

[0226] In some embodiments, the robot's grasping control device 300 also includes: an identification module, which is used to identify whether there is a graspable target object in the current environment based on the first perception data, and obtain an identification result; a second control module 302, which is also used to control the robotic arm to drive the second acquisition component to collect information on the target object when the recognition result indicates that there is a target object, and obtain second perception data.

[0227] In some embodiments, the third control module 303 is also used to determine a first recognition result corresponding to the first perception data and a second recognition result corresponding to the second perception data; based on the first recognition result and the second recognition result, control the robotic arm to grasp the target object.

[0228] In some embodiments, the first recognition result includes a first target category corresponding to the target object, and the second recognition result includes a second target category corresponding to the target object; the third control module 303 is also used to control the robotic arm to grasp the target object based on the first target grasping point corresponding to the target object when the first target category and the second target category are the same; the first target grasping point is identified from the first perception data and / or the second perception data based on the first target category; or, when the first target category and the second target category are different, determine the first target strategy to be executed to control the robotic arm to grasp the target object based on the first target strategy.

[0229] In some embodiments, the third control module 303 is also used to control the robotic arm to give up grasping the target object; control the robotic arm to grasp the target object based on the first target grasping point; the first target grasping point is obtained by identifying the first target category from the first perception data and / or the second perception data; control the robotic arm to grasp the target object based on the second target grasping point; the second target grasping point is obtained by identifying the second target category from the first perception data and / or the second perception data; control the robotic arm and / or the body to detour from the current first position to the second position of the target object, control the first acquisition component at the second position to collect information on the target object to obtain third perception data, and control the robotic arm to drive the second acquisition component to collect information on the target object to obtain fourth perception data, so as to control the robotic arm to grasp the target object based on the third perception data and the fourth perception data.

[0230] In some embodiments, the third control module 303 is also used to control the robotic arm to grasp the target object based on the third target grasping point or the fourth target grasping point when the third recognition result corresponding to the third perception data and the fourth recognition result corresponding to the fourth perception data are the same; the third target grasping point is identified from the third perception data and / or the fourth perception data based on the third target category in the third recognition result, and the fourth target grasping point is identified from the third perception data and / or the fourth perception data based on the fourth target category in the fourth recognition result; or, when the third recognition result and the fourth recognition result are different and the number of times the first acquisition component or the second acquisition component collects information on the target object exceeds a number threshold, control the robotic arm to grasp the target object based on the second target strategy, and the second target strategy includes one of the following: controlling the robotic arm to give up grasping the target object; controlling the robotic arm to grasp the target object based on the third target grasping point; controlling the robotic arm to grasp the target object based on the fourth target grasping point.

[0231] In some embodiments, the second control module 302 is also used to identify whether the target object meets the target condition based on the first candidate perception data currently collected by the second acquisition component; the target condition includes at least one of the following: the proportion of target perception data corresponding to the target object in the first candidate perception data is greater than or equal to a preset ratio, and the target object is located within the visual cone corresponding to the second acquisition component; when the target object meets the target condition, the first candidate perception data is used as the second perception data.

[0232] In some embodiments, the second control module 302 is also used to control the robotic arm and / or the fuselage to drive the second acquisition component to move and collect information on the target object when the target object does not meet the target conditions, so as to obtain second candidate perception data, so as to identify whether the target object meets the target conditions based on the second candidate perception data.

[0233] In some embodiments, the robotic arm is installed in a storage bin on top of the fuselage; the first control module 301 is also used to control the first acquisition component to collect first perception data in the current environment of the robot when the robotic arm is stored in the storage bin; the second control module 302 is also used to control the robotic arm to perform an out-of-bin operation, and control the robotic arm to drive the second acquisition component to collect information on the target object to obtain second perception data.

[0234] In some embodiments, the first control module 301 is also used to control the robot to perform a cleaning task when the robotic arm is retracted into the receiving bin; and to control the first acquisition component to collect first perception data in the current environment of the robot while the robot is performing the cleaning task.

[0235] Based on the above embodiment, the present application embodiment further provides a robot, Figure 4 A schematic diagram of the structure of a robot provided in an embodiment of the present application Figure 1 ,like Figure 4 As shown, the robot 400 includes: a body 410, a mechanical arm 420, a control component 430, a first acquisition component 440 and a second acquisition component 450, wherein:

[0236] A mechanical arm 420 is disposed on the top of the fuselage 410;

[0237] A first collection component 440 is disposed in front of the fuselage 410;

[0238] A second collection component 450 is disposed on the mechanical arm 420; and

[0239] The control component 430 is disposed inside the body 410. The control component 430:

[0240] Controlling the first acquisition component to collect first perception data in the current environment of the robot;

[0241] Control the robotic arm to drive the second acquisition component to collect second perception data in the current environment;

[0242] Based on the first perception data and the second perception data, the robotic arm is controlled to grasp a target object that can be grasped in the current environment.

[0243] In some embodiments, the control component further identifies whether there is a graspable target object in the current environment based on the first perception data, and obtains an identification result;

[0244] When the recognition result indicates that there is a target object, the control component controls the robotic arm to drive the second collection component to collect information on the target object to obtain second perception data.

[0245] In some embodiments, the control component further determines a first recognition result corresponding to the first sensed data, and a second recognition result corresponding to the second sensed data;

[0246] The control component also controls the robotic arm to grasp the target object based on the first recognition result and the second recognition result.

[0247] In some embodiments, the first recognition result includes a first target category corresponding to the target object, and the second recognition result includes a second target category corresponding to the target object; when the first target category and the second target category are the same, the control component further controls the robotic arm to grasp the target object based on a first target grasping point corresponding to the target object; the first target grasping point is obtained by recognizing the first perception data and / or the second perception data based on the first target category; or,

[0248] In the case where the first target category and the second target category are different, the control component further determines a first target strategy to be executed to control the robotic arm to grasp the target object based on the first target strategy.

[0249] In some embodiments, the control component further controls the robotic arm to abandon grasping the target object;

[0250] The control component also controls the mechanical arm to grasp the target object based on the first target grasping point; the first target grasping point is identified from the first perception data and / or the second perception data based on the first target category;

[0251] The control component also controls the robotic arm to grasp the target object based on the second target grasping point; the second target grasping point is identified from the first perception data and / or the second perception data based on the second target category;

[0252] The control component also controls the robotic arm and / or the fuselage to move from the current first position to the second position of the target object, controls the first acquisition component at the second position to collect information on the target object to obtain third perception data, and controls the robotic arm to drive the second acquisition component to collect information on the target object to obtain fourth perception data, so as to control the robotic arm to grasp the target object based on the third perception data and the fourth perception data.

[0253] In some embodiments, when the third recognition result corresponding to the third perception data and the fourth recognition result corresponding to the fourth perception data are the same, the control component further controls the robotic arm to grasp the target object based on the third target grasping point or the fourth target grasping point; the third target grasping point is identified from the third perception data and / or the fourth perception data based on the third target category in the third recognition result, and the fourth target grasping point is identified from the third perception data and / or the fourth perception data based on the fourth target category in the fourth recognition result; or,

[0254] When the third recognition result and the fourth recognition result are different and the number of times the first acquisition component or the second acquisition component collects information on the target object exceeds the number threshold, the control component further controls the manipulator to grasp the target object based on the second target strategy, and the second target strategy includes one of the following:

[0255] Control the robotic arm to give up grabbing the target object;

[0256] Controlling the robotic arm to grasp the target object based on the third target grasping point;

[0257] The robotic arm is controlled to grasp the target object based on the fourth target grasping point.

[0258] In some embodiments, the control component further identifies whether the target object meets the target condition based on the first candidate perception data currently collected by the second acquisition component; the target condition includes at least one of the following: the proportion of the target perception data corresponding to the target object in the first candidate perception data is greater than or equal to a preset ratio, and the target object is located in the visual cone corresponding to the second acquisition component;

[0259] When the target object meets the target condition, the control component also uses the first candidate perception data as the second perception data.

[0260] In some embodiments, when the target object does not meet the target conditions, the control component also controls the robotic arm and / or the fuselage to drive the second acquisition component to move and collect information on the target object to obtain second candidate perception data, so as to identify whether the target object meets the target conditions based on the second candidate perception data.

[0261] In some embodiments, the robotic arm is installed in a storage compartment on top of the fuselage; when the robotic arm is stored in the storage compartment, the control component further controls the first acquisition component to acquire first perception data in the current environment where the robot is located;

[0262] The control component also controls the robotic arm to perform an exit operation, and controls the robotic arm to drive the second acquisition component to collect information on the target object to obtain second perception data.

[0263] Figure 5 A schematic diagram of the structure of a robot provided in an embodiment of the present application Figure 2 ,like Figure 5 As shown, the robot 400 includes: a body 410, a mechanical arm 420, a control component 430, a first collection component 440, a second collection component 450 and a storage bin 460, wherein: the mechanical arm 420 is arranged in the storage bin 460 on the top of the body 410, the first collection component 440 is arranged in front of the body 410, and the second collection component 450 is arranged in the mechanical arm 420. The control component 430 is arranged inside the body 410.

[0264] In some embodiments, when the robotic arm is retracted into the receiving bin, the control component further controls the robot to perform a cleaning task;

[0265] When the robot performs a cleaning task, the control component also controls the first acquisition component to acquire first perception data in the current environment where the robot is located.

[0266] Fig. 6A A schematic diagram of the structure of a robot provided in an embodiment of the present application Figure 3 ,like Fig. 6A As shown, the robot 400 includes: a body 410, a mechanical arm, a control component, a first collection component 440, a second collection component and a storage bin 460, wherein: the mechanical arm is arranged in the storage bin 460 at the top of the body 410, the first collection component 440 is arranged in front of the body 410, and the second collection component is arranged in the mechanical arm. The control component is arranged inside the body 410. As can be seen from the figure, the mechanical arm is put into the storage bin 460. At this time, the robot 400 is in the process of performing the cleaning task, and the control component can control the first collection component 440 to collect the first perception data in the current environment where the robot 400 is located.

[0267] Figure 6B A schematic diagram of the structure of a robot provided in an embodiment of the present application Figure 4 ,like Figure 6BAs shown, the robot 400 includes: a body 410, a mechanical arm 420, a control component 430, a first collection component 440, a second collection component 450 and a storage bin 460, wherein: the mechanical arm 420 is arranged in the storage bin 460 on the top of the body 410, the first collection component 440 is arranged in front of the body 410, and the second collection component 450 is arranged in the mechanical arm 420. The control component is arranged inside the body 410. As can be seen from the figure, the mechanical arm 420 performs the out-of-bin operation and extends out of the storage bin 460, and the mechanical arm 420 can be controlled to drive the second collection component 450 to collect information on the target object to obtain the second perception data.

[0268] The description of the above robot and robot grasping control device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the robot and robot grasping control device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0269] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0270] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a ROM, a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0271] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0272] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and the computer program implements the above method when executed by a processor. The computer-readable storage medium may be transient or non-transient.

[0273] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

[0274] An embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements some or all of the steps in the above-mentioned robot grasping control method.

[0275] An embodiment of the present application provides a processor, which is communicatively connected to a memory, and the memory stores a computer program that can be run on the processor. When the processor executes the computer program, the steps in the above-mentioned robot grasping control method are implemented.

[0276] An embodiment of the present application provides a robot, including a processor and a memory, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps in the above-mentioned robot grasping control method are implemented.

[0277] It should be noted here that the description of the above robot, storage medium, device and program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the robot, storage medium, device and program product embodiments of this application, please refer to the description of the method embodiment of this application for understanding.

[0278] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0279] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0280] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0281] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0282] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0283] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, read-only memories, magnetic disks or optical disks.

[0284] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0285] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A robot grasping control method, characterized in that: The robot comprises a body, a mechanical arm arranged on the top of the body, a first collection component arranged on the body, and a second collection component arranged on the mechanical arm, and the method comprises: Controlling the first acquisition component to collect first perception data in the current environment where the robot is located; Controlling the robotic arm to drive the second acquisition component to collect second perception data in the current environment; Based on the first perception data and the second perception data, the robotic arm is controlled to grasp a graspable target object in the current environment.

2. The grasping control method according to claim 1, characterized in that: The controlling the mechanical arm to drive the second acquisition component to collect second perception data in the current environment includes: According to the first sensing data, identifying whether there is a graspable target object in the current environment, and obtaining a recognition result; When the recognition result indicates that the target object exists, the robotic arm is controlled to drive the second acquisition component to collect information from the target object to obtain second perception data.

3. The grasping control method according to claim 1, characterized in that: The controlling the robotic arm to grasp a graspable target object in the current environment based on the first perception data and the second perception data includes: Determine a first recognition result corresponding to the first perception data and a second recognition result corresponding to the second perception data; Based on the first recognition result and the second recognition result, the robotic arm is controlled to grasp the target object.

4. The grasping control method according to claim 3, characterized in that: The first recognition result includes a first target category corresponding to the target object, and the second recognition result includes a second target category corresponding to the target object; The controlling the robot arm to grasp the target object based on the first recognition result and the second recognition result includes: In the case where the first target category and the second target category are the same, the robotic arm is controlled to grasp the target object based on a first target grasping point corresponding to the target object; the first target grasping point is identified from the first perception data and / or the second perception data based on the first target category; or In a case where the first target category and the second target category are different, a first target strategy to be executed is determined to control the robotic arm to grasp the target object based on the first target strategy.

5. The grasping control method according to claim 4, characterized in that: The controlling the robotic arm to grasp the target object based on the first target strategy includes one of the following: Controlling the robotic arm to give up grasping the target object; Controlling the robotic arm to grasp the target object based on a first target grasping point; the first target grasping point is identified from the first perception data and / or the second perception data based on the first target category; Controlling the robotic arm to grasp the target object based on a second target grasping point; the second target grasping point is identified from the first perception data and / or the second perception data based on the second target category; Control the robotic arm and / or the body to move from the current first position to the second position of the target object, control the first acquisition component to collect information on the target object at the second position to obtain third perception data, and control the robotic arm to drive the second acquisition component to collect information on the target object to obtain fourth perception data, so as to control the robotic arm to grasp the target object based on the third perception data and the fourth perception data.

6. The grasping control method according to claim 5, characterized in that: The controlling the robotic arm to grasp the target object based on the third sensing data and the fourth sensing data includes: When the third recognition result corresponding to the third perception data and the fourth recognition result corresponding to the fourth perception data are the same, the robotic arm is controlled to grasp the target object based on the third target grasping point or the fourth target grasping point; the third target grasping point is obtained by recognizing from the third perception data and / or the fourth perception data based on the third target category in the third recognition result, and the fourth target grasping point is obtained by recognizing from the third perception data and / or the fourth perception data based on the fourth target category in the fourth recognition result; or, When the third recognition result is different from the fourth recognition result and the number of times the first acquisition component or the second acquisition component collects information on the target object exceeds a number threshold, the robotic arm is controlled to grasp the target object based on a second target strategy, and the second target strategy includes one of the following: Controlling the robotic arm to give up grasping the target object; Controlling the robotic arm to grasp the target object based on the third target grasping point; The robotic arm is controlled to grasp the target object based on the fourth target grasping point.

7. The grasping control method according to claim 2, characterized in that: The controlling the mechanical arm to drive the second acquisition component to collect information from the target object to obtain second perception data includes: Based on the first candidate perception data currently collected by the second acquisition component, identifying whether the target object meets the target condition; the target condition includes at least one of the following: the proportion of the target perception data corresponding to the target object in the first candidate perception data is greater than or equal to a preset ratio, and the target object is located in the visual cone corresponding to the second acquisition component; When the target object satisfies the target condition, the first candidate perception data is used as the second perception data.

8. The grasping control method according to claim 7, characterized in that: The method further comprises: When the target object does not meet the target condition, the robotic arm and / or the fuselage is controlled to drive the second acquisition component to move and collect information on the target object to obtain second candidate perception data, so as to identify whether the target object meets the target condition based on the second candidate perception data.

9. The grasping control method according to any one of claims 2 to 8, characterized in that: The robot arm is installed in the storage compartment on the top of the fuselage; the controlling the first acquisition component to collect first perception data in the current environment where the robot is located includes: When the robot arm is put into the storage bin, controlling the first acquisition component to acquire the first perception data in the current environment where the robot is located; The controlling the mechanical arm to drive the second acquisition component to collect information from the target object to obtain second perception data includes: The robotic arm is controlled to perform an exit operation, and the robotic arm is controlled to drive the second acquisition component to collect information on the target object to obtain the second perception data.

10. The grasping control method according to claim 9, characterized in that: When the robot arm is received in the storage bin, controlling the first acquisition component to acquire first perception data in the current environment where the robot is located includes: When the robotic arm is retracted into the receiving bin, controlling the robot to perform a cleaning task; During the process of the robot performing the cleaning task, the first acquisition component is controlled to acquire the first perception data in the current environment where the robot is located.

11. A robot grasping control device, characterized in that: The robot comprises a body, a mechanical arm arranged on the top of the body, a first collection component arranged on the body, and a second collection component arranged on the mechanical arm, and the device comprises: A first control module, used to control the first acquisition component to collect first perception data in the current environment where the robot is located; A second control module, used for controlling the mechanical arm to drive the second acquisition component to collect second perception data in the current environment; The third control module is used to control the robotic arm to grasp a graspable target object in the current environment based on the first perception data and the second perception data.

12. The gripping control device according to claim 11, characterized in that: The device also includes: A recognition module, used to recognize whether there is a graspable target object in the current environment according to the first sensing data, and obtain a recognition result; The second control module is further used to control the robotic arm to drive the second acquisition component to collect information from the target object to obtain second perception data when the recognition result indicates that the target object exists.

13. The gripping control device according to claim 11, characterized in that: The third control module is also used to determine a first recognition result corresponding to the first perception data and a second recognition result corresponding to the second perception data; based on the first recognition result and the second recognition result, control the robotic arm to grasp the target object.

14. The gripping control device according to claim 13, characterized in that: The first recognition result includes a first target category corresponding to the target object, and the second recognition result includes a second target category corresponding to the target object; The third control module is further configured to control the robotic arm to grasp the target object based on a first target grasping point corresponding to the target object when the first target category and the second target category are the same; The first target grasping point is identified from the first perception data and / or the second perception data based on the first target category; or, when the first target category and the second target category are different, a first target strategy to be executed is determined to control the robotic arm to grasp the target object based on the first target strategy.

15. The gripping control device according to claim 14, characterized in that: The third control module is further used to control the robotic arm to give up grasping the target object; control the robotic arm to grasp the target object based on a first target grasping point; the first target grasping point is identified from the first perception data and / or the second perception data based on the first target category; The robotic arm is controlled to grasp the target object based on a second target grasping point; the second target grasping point is identified from the first perception data and / or the second perception data based on the second target category; the robotic arm and / or the fuselage is controlled to detour from the current first position to the second position of the target object, and at the second position, the first acquisition component is controlled to collect information from the target object to obtain third perception data, and the robotic arm is controlled to drive the second acquisition component to collect information from the target object to obtain fourth perception data, so as to control the robotic arm to grasp the target object based on the third perception data and the fourth perception data.

16. The gripping control device according to claim 15, characterized in that: The third control module is further used to control the robotic arm to grasp the target object based on the third target grasping point or the fourth target grasping point when the third recognition result corresponding to the third perception data and the fourth recognition result corresponding to the fourth perception data are the same; the third target grasping point is obtained by identifying the third perception data and / or the fourth perception data based on the third target category in the third recognition result, and the fourth target grasping point is obtained by identifying the third perception data and / or the fourth perception data based on the fourth target category in the fourth recognition result; or, when the third recognition result and the fourth recognition result are different and the number of times the first acquisition component or the second acquisition component collects information on the target object exceeds a number threshold, control the robotic arm to grasp the target object based on a second target strategy, and the second target strategy includes one of the following: controlling the robotic arm to give up grasping the target object; controlling the robotic arm to grasp the target object based on the third target grasping point; controlling the robotic arm to grasp the target object based on the fourth target grasping point.

17. The gripping control device according to claim 12, characterized in that: The second control module is also used to identify whether the target object meets the target condition based on the first candidate perception data currently collected by the second acquisition component; the target condition includes at least one of the following: the proportion of the target perception data corresponding to the target object in the first candidate perception data is greater than or equal to a preset ratio, and the target object is located in the visual cone corresponding to the second acquisition component; when the target object meets the target condition, the first candidate perception data is used as the second perception data.

18. The gripping control device according to claim 17, characterized in that: The second control module is also used to control the robotic arm and / or the fuselage to drive the second acquisition component to move and collect information on the target object when the target object does not meet the target condition, so as to obtain second candidate perception data, so as to identify whether the target object meets the target condition based on the second candidate perception data.

19. The gripping control device according to any one of claims 12 to 18, characterized in that: The mechanical arm is installed in the accommodation compartment on the top of the fuselage; The first control module is further used to control the first acquisition component to collect the first perception data in the current environment where the robot is located when the robot arm is stored in the accommodation bin; The second control module is further used to control the robotic arm to perform an exit operation, and control the robotic arm to drive the second acquisition component to collect information on the target object to obtain the second perception data.

20. The gripping control device according to claim 19, characterized in that: The first control module is further used to control the robot to perform a cleaning task when the robotic arm is retracted into the containing bin; and to control the first acquisition component to collect the first perception data in the current environment of the robot during the robot performing the cleaning task.

21. A robot, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, the steps in the method according to any one of claims 1 to 10 are implemented.

22. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 10 are implemented.

23. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images