Ground small object grabbing method and system of mechanical arm with obstacle avoidance capability

Abstract

A mechanical arm ground small object grabbing method and system with obstacle avoidance capability, the method comprising: S1: arranging sensors, collecting point clouds for environment modeling; S2: carrying out noise filtering on the point cloud; S3: generating a hypothetical volume of the obstacle blocking part on the point cloud; S4: filtering the obstacles around the target on the point cloud; S5: force feedback control of mechanical arm obstacle avoidance; S6: object detection and end posture calculation; S7: planning path to execute grabbing. The present application is used when the mechanical arm grabs small-sized objects on hard ground, combines sensor technology, environment modeling, obstacle avoidance algorithm and grabbing strategy, so that the mechanical arm can autonomously avoid obstacles in a complex environment and accurately grab small objects on the ground.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot control and machine vision, in particular to a mechanical arm ground small object grasping method and system with obstacle avoidance capability. BACKGROUND

[0002] In practical applications, such as cleaning robots, logistics and other scenarios, the mechanical arm needs to accurately grasp the ground small object, and when encountering obstacles, it can autonomously avoid obstacles to perform subsequent processing and transportation tasks. However, in a complex environment, the obstacle avoidance and force feedback control of the mechanical arm are often affected by multiple factors, which requires new methods to solve.

[0003] Ground small object grasping research faces some challenges. First, due to the diversity of size and shape of ground small objects, the grasping difficulty of the mechanical arm is relatively large, and when encountering obstacles, the mechanical arm needs to autonomously avoid obstacles to avoid collision. Secondly, the force feedback control of the mechanical arm needs to analyze and process the moving obstacles to ensure the stability and safety of the grasping process. SUMMARY

[0004] The present application overcomes the above-mentioned shortcomings of the prior art, a method for autonomous obstacle avoidance and target grasping of a mechanical arm.

[0005] To solve the above problems, the present application proposes a ground small object mechanical arm obstacle avoidance and force feedback control grasping method. This method uses deep learning and computer vision technologies to identify and analyze ground small objects, and uses different sensors to obtain environmental information to achieve obstacle avoidance and force feedback control of the mechanical arm. In addition, the present application also uses a force controller to control the grasping process of the mechanical arm to ensure the stability and safety of the grasping.

[0006] A mechanical arm ground small object grasping method with obstacle avoidance capability, characterized in that it comprises:

[0007] S1: arranging sensors on the robot system to collect point clouds for environment modeling;

[0008] S2: filtering out noise points on the point cloud;

[0009] S3: generating a hypothetical volume of the obstacle occlusion part on the point cloud;

[0010] S4: filtering out obstacles around the target on the point cloud;

[0011] S5: generating obstacle voxel units;

[0012] S6: force feedback control of the mechanical arm obstacle avoidance;

[0013] S7: object detection and end effector pose calculation;

[0014] S8: robot arm path planning and execution of grasping.

[0015] Further, step S1 is specifically:

[0016] The robot system includes two front depth cameras, namely left and right eye cameras, a robot arm hand-eye camera, and a robot arm. The left and right eye cameras are symmetrically distributed and fixed on the robot. The robot arm is fixed on the robot through a base and is fixed relative to the left and right eye cameras. The hand-eye camera is fixed to the palm joint at the end of the robot arm and moves with the robot arm end.

[0017] Further, step S2 is specifically:

[0018] S21. Use the point cloud output by the two front depth cameras as the information source for obstacle avoidance perception.

[0019] S22. First, uniformly downsample the point cloud to about 4cm interval. Then, with a search radius of 10cm, filter out the outliers whose point cloud number is less than 6 points within the radius.

[0020] Further, step S3 is specifically:

[0021] First, capture all point clouds above 5cm from the ground. Through outlier filtering, obtain the point cloud of the obstacle edge. Then, translate the edge backward to generate a hypothetical volume. Finally, generate the obstacle voxel unit.

[0022] Further, step S4 is specifically:

[0023] Filter out the ground within the surrounding area of the grasped object, including: generate a rectangle with a front and back length of 30cm, a left and right length of 30cm, and an up and down length of 8cm with the stone center as the center point. Remove the collision voxel units within the rectangular frame.

[0024] Further, step S5 is specifically:

[0025] Convert the point cloud data to voxel representation, including the following steps:

[0026] Determine the size of the voxel: First, determine the size of the voxel, i.e. the edge length of each voxel. The choice of edge length depends on the application requirements and the resolution of the point cloud data.

[0027] Create a voxel grid: According to the determined voxel size, create a voxel grid in three-dimensional space.

[0028] The voxel grid is a three-dimensional array composed of several voxel units.

[0029] • Map the point cloud data to a voxel grid: Traverse each point in the point cloud and map it to the corresponding voxel cell based on the position of the point. The nearest neighbor method or interpolation method can be used to determine the position of the point in the voxel.

[0030] Further, step S6 is specifically:

[0031] By higher frequency of mechanical arm joint force feedback, when the mechanical arm collides with the obstacle, terminate the current trajectory in time, return to the previous position, and re-plan the mechanical arm path;

[0032] The method of detecting collision is to monitor the force feedback of each joint during normal sampling for a period of time, and then extend 2N / m of bending moment in positive and negative directions as the normal mechanical arm joint force feedback interval. If it exceeds this interval, the mechanical arm movement is terminated.

[0033] Further, step S7 is specifically:

[0034] By using the pre-positioned depth camera, the target is recognized and positioned by the convolutional neural network yolox, and the position information of the target is obtained. According to the position information of the target, the end pose of the mechanical arm is calculated. When collecting small objects on the ground, the following factors need to be considered in the calculation of the end pose of the kinova gen2 two-finger mechanical arm:

[0035] · The double finger is parallel to the ground to prevent the first finger from touching the ground when grabbing;

[0036] · The arm is fully extended close to a straight line at the farthest point, and the grabbing range is larger relative to the vertical ground;

[0037] · When far away, the claw is inclined outward like Figure 14 When close, the claw is inclined inward;

[0038] · When far away, the claw is straight towards the arm base and stretched far;

[0039] · When close, the double finger is parallel to the machine body to increase the grabbing range when close without colliding with the machine body;

[0040] Therefore, the end pose can be associated with the position of the target to be grabbed. Assuming that the target is located at the position x, y, z in the mechanical arm base coordinate system, the base coordinate system is x forward, y to the side, and z upward.

[0041] Use python language to call scipy library to calculate the end pose:

[0042] from scipy.spatial.transform import Rotation as R

[0043] First, the end is calculated as a positive downward attitude as an adjustment reference attitude, at this time the end coordinate system z axis is downward, the x axis is sideways, and the y axis is forward.

[0044] r1=R.from_euler('ZYX',[-3.1415 / 2,3.1415,0])

[0045] With the end being positive downward as the reference coordinate system, the end attitude is adjusted according to the target position according to the above principle.

[0046] First, in order to ensure that the claw is facing the arm base when the end is away, and the double fingers are parallel to the machine body when close, the end is rotated around the Z axis by an angle of:

[0047] Theta=atan(y / x)*(x-0.2))*-3

[0048] In order to ensure that the claw is outwardly inclined when the end is away from the machine body, and the claw is inwardly inclined when close, the end is rotated around the X axis by an angle of:

[0049] Phi=(sqrt(x**2+y**2)-0.4)*-3

[0050] In order to ensure that the end is always parallel to the ground, the end is rotated around the Y axis by an angle of 0 degrees.

[0051] Therefore, the end attitude is described as follows:

[0052] r2=R.from_euler('ZYX',[(Theta,0,Phi])

[0053] Finally, the end attitude under the base coordinate system is calculated.

[0054] r=r1*r2

[0055] Further, step S8 is specifically:

[0056] Use moveit as a robot planner, set the target pose to determine the target pose of the robot when grabbing, use the planner provided by MoveIt to generate the path of the robot by taking the target pose as input. Different planning algorithms and parameters can be selected to meet specific needs; finally, the generated path is sent to the robot controller to execute the grabbing action. Use the interface provided by MoveIt or write custom control logic to implement path execution.

[0057] The second aspect of the application relates to a mechanical arm ground small object grabbing system with obstacle avoidance capability, comprising:

[0058] An environment modeling module is used to arrange sensors, collect point clouds for environment modeling.

[0059] A noise removal module is used to remove noise from the point cloud.

[0060] a hypothetical volume production module of the obstacle blocking part, for generating a hypothetical volume of the obstacle blocking part on the point cloud;

[0061] a target surrounding obstacle filtering module, for filtering the target surrounding obstacle on the point cloud;

[0062] an obstacle voxel unit production module, for generating an obstacle voxel unit;

[0063] a force feedback control module, for force feedback control of the robot arm obstacle avoidance;

[0064] an object detection and grasping end pose calculation module, for object detection and grasping end pose calculation;

[0065] a robot arm path planning and grasping execution module, for robot arm path planning and grasping execution.

[0066] The third aspect of the application relates to a robot arm ground small object grasping device with obstacle avoidance capability, comprising a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to implement the robot arm ground small object grasping method with obstacle avoidance capability.

[0067] The fourth aspect of the application relates to a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the robot arm ground small object grasping method with obstacle avoidance capability.

[0068] The working principle of the application is:

[0069] The grasping target and obstacles in the environment are perceived by the depth camera, the robot arm trajectory is planned according to the target position and obstacle position through the algorithm, the robot arm grasps the object, and in the process of robot arm movement, the current action is terminated in time according to the joint torque information perceived by the force feedback sensor, and the robot arm passive obstacle avoidance is realized. According to the target position, the end grasping pose is generated through the algorithm, the robot arm trajectory is planned according to the grasping pose, and the obstacle avoidance grasping is realized.

[0070] The innovation of the application is:

[0071] For the point cloud generated by the depth camera, efficient noise filtering is carried out, the obstacle shielding part volume that the camera cannot observe is generated through the self-developed algorithm, and the obstacles around the target affecting planning are filtered out, so that a more stable visual obstacle avoidance planning scheme is realized, and combined with the force feedback sensor of the mechanical arm, the autonomous force perception and avoidance after the mechanical arm collides with the obstacle are realized. Through the combination of the two obstacle avoidance schemes, a high-stability mechanical arm obstacle avoidance scheme is realized. Through the autonomous algorithm, the mechanical arm end gripping pose is generated according to the gripping target position, so that a larger gripping range is realized, and compared with the deep learning calculation mechanical arm end pose, the stability and calculation efficiency are higher.

[0072] The advantages of the present application are:

[0073] By combining visual active obstacle avoidance and force feedback passive obstacle avoidance, high-stability obstacle avoidance capability in the process of mechanical arm gripping objects is realized. A larger mechanical arm gripping range is realized through the algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 It is a structural schematic diagram of the robot used in the present application.

[0075] Figure 2 It is a schematic diagram of the robot coordinate of the present application.

[0076] Figure 3 It is a scene arrangement schematic diagram of the present application.

[0077] Figure 4 It is a schematic diagram of the robot point cloud collection of the present application.

[0078] Figure 5 It is a point cloud filtering schematic diagram of the present application.

[0079] Figure 6 It is a schematic diagram of the robot point cloud collection of the present application.

[0080] Figure 7 It is a schematic diagram of the robot capturing the point cloud above 5cm of the ground of the present application.

[0081] Figure 8 It is a schematic diagram of the point cloud of the obstacle edge obtained by outlier filtering of the present application.

[0082] Figure 9 It is a schematic diagram of the present application, which translates the edge to the rear to generate a hypothetical volume, and finally generates the obstacle voxel unit.

[0083] Figure 10 It is an effect diagram of the present application for removing the collision voxel unit in the rectangular frame.

[0084] Figure 11 It is an effect diagram of the present application for converting the point cloud into a voxel unit.

[0085] Figure 12 is the target positioning and grasping posture diagram of the application.

[0086] Figure 13 is the mechanical arm near-grasping posture diagram of the robot of the application, and the yellow color is a virtual obstacle.

[0087] Figure 14 is the mechanical arm far-grasping posture diagram of the robot of the application, and the yellow color is a virtual obstacle.

[0088] Figure 15 is the mechanical arm grasping posture diagram of the robot of the application.

[0089] Figure 16 is the method flowchart of the application.

[0090] Figure 17 is the system structure diagram of the application. DETAILED DESCRIPTION

[0091] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the application.

[0092] It should be noted that the features in the following embodiments and implementation manners can be combined with each other without conflict.

[0093] Embodiment 1

[0094] The application realizes ground small object grasping through a camera and a mechanical arm of a robot. A hexapod robot is taken as an illustrative object, but the application is not limited to this type of robot. When a target object appears in front of the robot, the embodiment provides a method for autonomous obstacle avoidance and target grasping of a mechanical arm. The specific steps are as follows:

[0095] 1. A sensor is arranged on the robot system to collect point clouds for environment modeling;

[0096] The robot system involved in the application includes two front depth cameras (left eye camera and right eye camera), a mechanical arm hand-eye camera, and a mechanical arm. The positional relationship between the camera and the mechanical arm is as shown in Figure 1 , and the coordinate system relationship is as shown in Figure 2 . First, a scene is built as a scene description, and the scene where the robot is located is as shown in Figure 3 . Point clouds are collected through the two front cameras, and the collected point clouds are as shown in Figure 4 .

[0097] 2. Noise filtering on point cloud;

[0098] The hexapod robot's grabbing range is within 0.5m in front. Due to the large error of radar in short distance, the front blind-filling radar cannot provide effective information in this area, so the point cloud output by two front depth cameras is used as the information source for obstacle avoidance. The point cloud provided by the depth camera will have noise, as shown in Fig. 2, and noise is easily mistaken for obstacles in the obstacle avoidance process, so noise needs to be filtered out. Figure 4

[0099] The noise of the depth camera is discrete or small, so this paper first uniformly down-samples the point cloud to about 4cm apart, and then filters out the outliers of the point cloud with less than 6 points within a radius of 10cm. The effect of the point cloud after filtering outliers is shown in Fig. 3. This method is efficient and has good filtering effect. Figure 5

[0100] 3. Generating the hypothetical volume of the obstacle's occluded part on the point cloud;

[0101] The height of the front camera is about 30cm above the ground, so for obstacles of a certain height, only the point cloud in the front direction can be collected, as shown in Fig. 4, and the point cloud on the top of the obstacle cannot be collected. In obstacle avoidance, the robot arm may collide with the top of the obstacle, so a hypothetical volume needs to be generated for the part of the obstacle that cannot be observed. For safety reasons, this paper adopts a relatively conservative method, assuming that all obstacles detected above the ground within a 5cm interval have a thickness of 30cm behind the obstacle. Therefore, this part of the hypothetical point cloud needs to be generated. All point clouds can be directly translated a few layers back, but since the robot arm needs to avoid all obstacle volumes in collision detection, the more obstacle voxel units, the more computation. This method will generate a large number of invalid collision voxel units. The method used in this paper is to first capture all point clouds above 5cm, as shown in Fig. 5, and obtain the point cloud of the edge of the obstacle through outlier filtering, as shown in Fig. 6, and then translate the edge backward to generate the hypothetical volume. The final generated obstacle voxel unit is shown in Fig. 7. This way greatly reduces the computation. Figure 6 Figure 7 Figure 8 Figure 9 Figure 8

[0102] 4. Filtering obstacles around the target on the point cloud;

[0103] ​​​​​​​Since the point cloud generated by the obstacle voxel unit includes the grabbed object itself and the ground, in order to enable the robot arm to approach the ground to grab the object, when generating obstacles, the grabbed object itself and the ground within a certain interval need to be filtered out. The method adopted in this paper is to generate a rectangle with a length of 30 cm in front and back, a length of 30 cm in left and right, and a length of 8 cm in up and down, with the center of the stone as the center point, and remove the collision voxel units within the rectangular frame, as shown in Figure 10 .

[0104] 5. Generating obstacle voxel units;

[0105] A voxel is a discrete unit in three-dimensional space, similar to the role of a pixel in a two-dimensional image. By converting point cloud data into voxel representation, further processing and analysis can be facilitated.

[0106] The process of converting point cloud to voxel includes the following steps:

[0107] • Determine the size of the voxel: First, the size of the voxel, i.e. the edge length of each voxel, needs to be determined. The choice of edge length depends on the application requirements and the resolution of the point cloud data.

[0108] • Create a voxel grid: According to the determined size of the voxel, create a voxel grid in three-dimensional space.

[0109] The voxel grid is a three-dimensional array composed of several voxel units.

[0110] • Map point cloud data to voxel grid: Traverse each point in the point cloud, and map it to the corresponding voxel unit according to the position of the point. Nearest neighbor method or interpolation method can be used to determine the position of the point in the voxel.

[0111] By converting point cloud to voxel, point cloud data can be converted to regular voxel representation, facilitating voxel-level operations and analysis such as voxel reconstruction, voxel classification, etc. At the same time, due to the lower dimension of voxel representation, storage space and computing resources can be saved. The effect of converting point cloud to voxel is shown in Figure 11 .

[0112] 6. Force feedback control of robot arm obstacle avoidance;

[0113] Through visual detection for obstacle avoidance, the robot arm grabbing has a part of obstacle avoidance ability, but the visual detection has high cost of computing power, therefore, the method adopted in this paper is to generate visual observation obstacles only once at the same position, and when the visual detection frequency is low, there is a possibility that moving objects block and cause avoidance not in time. Therefore, further through higher frequency of robot arm joint force feedback, when the robot arm collides with obstacles, terminate the current trajectory in time, return to the previous position, and re-plan the robot arm path.

[0114] The control range of the force feedback of each joint of the mechanical arm needs to meet the normal use, that is, the joint force is within the control range, and when encountering an obstacle, the collision is detected as soon as possible without damaging the mechanical arm itself. The method used in this paper is to monitor the force feedback of each joint during a period of normal sampling process, and then expand the bending moment of 2N / m in the positive and negative directions as the normal mechanical arm joint force feedback range, and the range exceeding the range will terminate the movement of the mechanical arm.

[0115] 7. Object detection and end pose calculation;

[0116] The target position information is obtained by recognizing and positioning the target through the front depth camera and the convolutional neural network yolox. Further, the end pose of the mechanical arm is calculated according to the position information of the target. In this paper, the kinova gen2 two-finger mechanical arm is used, and when collecting small objects on the ground, the following factors need to be considered in the end pose calculation:

[0117] · The double finger is parallel to the ground to prevent the first finger from touching the ground when grabbing

[0118] · The arm is fully extended at the farthest point to approach a straight line, and the grabbing range is larger relative to the vertical ground, such as Figure 15

[0119] · The claw is inclined outward when it is far away, such as Figure 14 , and the claw is inclined inward when it is close, such as Figure 13

[0120] · The claw is directly facing the arm base when it is far away, and it is stretched far away, such as Figure 14

[0121] · The double finger is parallel to the machine body when it is close, which increases the grabbing range when it is close, and avoids collision with the machine body,

[0122] such as Figure 15

[0123] Therefore, the end pose can be associated with the position of the target to be grabbed, assuming that the target is located at x, y, z in the base coordinate system (tf_2) of the mechanical arm base coordinate system, and the base coordinate system is x forward, y to the side, and z upward, as shown in Figure 2 . Figure 15

[0124] This paper uses python language to call scipy library to calculate the end pose:

[0125] from scipy.spatial.transform import Rotation as R

[0126] First, calculate the end pose that is directly downward, which is used as the reference pose for adjustment, such as Figure 15 ​Yellow posture, at this time the end coordinate system z axis is downward, x axis is lateral, y axis is forward.

[0127] r1=R.from_euler('ZYX',[-3.1415 / 2,3.1415,0])

[0128] Take the end pointing straight down as the reference coordinate system, according to the target position, adjust the end posture according to the above principles.

[0129] First, in order to ensure that the claw is facing the arm base when the end is away, and the double fingers are parallel to the body when close, the end rotates around the Z axis at an angle of:

[0130] Theta=atan(y / x)*(x-0.2))*-3

[0131] In order to ensure that the claw is outwardly inclined when the end is away from the body, and the claw is inwardly inclined when close, the end rotates around the X axis at an angle of:

[0132] Phi=(sqrt(x**2+y**2)-0.4)*-3

[0133] In order to ensure that the end is always parallel to the ground, the end rotates around the Y axis at an angle of 0 degrees.

[0134] Therefore, the end posture is described as follows:

[0135] r2=R.from_euler('ZYX',[(Theta,0,Phi])

[0136] Finally, the end posture relative to the base coordinate system is calculated.

[0137] r=r1*r2

[0138] 8. The mechanical arm plans the path and executes the grabbing.

[0139] The application adopts moveit as the mechanical arm planner, sets the target pose, determines the target pose of the mechanical arm when grabbing, further uses the planner provided by MoveIt, takes the target pose as input, generates the path of the mechanical arm. Different planning algorithms and parameters can be selected to meet specific needs. Finally, the generated path is sent to the mechanical arm controller to execute the grabbing action. The interface provided by MoveIt or custom control logic can be used to implement path execution.

[0140] Example 2

[0141] This embodiment relates to a mechanical arm ground small object grabbing system with obstacle avoidance capability, which is used to realize the mechanical arm ground small object grabbing method with obstacle avoidance capability in example 1, comprising:

[0142] An environment modeling module is arranged to arrange sensors, and collect point clouds for environment modeling.

[0143] A noise filtering module is arranged to filter noise on the point clouds.

[0144] A hypothetical volume production module of the obstacle blocking part is arranged to generate a hypothetical volume of the obstacle blocking part on the point clouds.

[0145] A target surrounding obstacle filtering module is arranged to filter the target surrounding obstacle on the point clouds.

[0146] An obstacle voxel unit production module is arranged to generate an obstacle voxel unit.

[0147] A force feedback control module is arranged to perform force feedback control for the robot arm obstacle avoidance.

[0148] An object detection and grasping end pose calculation module is arranged to calculate the object detection and grasping end pose.

[0149] A robot arm path planning and grasping execution module is arranged to plan the path of the robot arm and execute the grasping.

[0150] Embodiment 3

[0151] This embodiment relates to a computer readable storage medium, which stores a program, and the program is executed by a processor to realize a robot arm ground small object grasping method with obstacle avoidance capability.

[0152] Embodiment 4

[0153] This embodiment relates to a robot arm ground small object grasping device with obstacle avoidance capability, which comprises a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to realize the robot arm ground small object grasping method with obstacle avoidance capability of embodiment 1.

[0154] At the hardware level, the robot arm ground small object grasping device with obstacle avoidance capability comprises a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course, other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs. Of course, in addition to the software implementation, the present application does not exclude other implementation modes, such as logic devices or software and hardware combined mode, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but also can be hardware or logic device.

[0155] For a technical improvement, it can be obvious whether the improvement is in hardware (e.g., improvement of circuit structures of diodes, transistors, switches, etc.) or in software (e.g., improvement of method flow). However, with the development of technology, many improvements of method flow today can be considered as direct improvements of hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement of method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A designer programs a digital system "integrated" on a PLD by himself / herself, without having to ask a chip manufacturer to design and manufacture a special integrated circuit chip. Moreover, instead of manually manufacturing an integrated circuit chip, such programming is now mostly implemented by "logic compiler" software, which is similar to a software compiler used when developing programs, and the original code before compilation also needs to be written in a specific programming language, which is called a hardware description language (HDL), and there are many kinds of HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. It should be clear to those skilled in the art that only a little logical programming of the method flow in the above-mentioned hardware description languages and programming into an integrated circuit can easily obtain a hardware circuit that implements the logical method flow.

[0156] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0157] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0158] For the sake of description, the above apparatuses are described in various units with functions respectively. Of course, the functions of the units can be implemented in one or more software and / or hardware in the implementation of the present application.

[0159] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0160] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0161] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0162] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0163] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0164] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory. The memory is an example of computer-readable media.

[0165] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0166] It should also be noted that the terms "comprising", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0167] Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system or computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0168] The present application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The present application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0169] The various embodiments in the present application are described in a progressive manner, and the same or similar parts among the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0170] The above merely describes the embodiments of the present application, and is not intended to limit the present application. The present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. A method for gripping small objects on the ground using a robotic arm with obstacle avoidance capabilities, characterized in that, include: S1: Deploy sensors on the robot system to collect point cloud data for environmental modeling; S2: Noise removal on point cloud; S3: Generate the imaginary volume of the occluded part of the obstacle on the point cloud; specifically: first capture all point clouds above 5cm on the ground, filter out outliers to obtain the point cloud of the obstacle edge, and then translate the edge backward to generate the imaginary volume. S4: Filter out obstacles around the target in the point cloud; S5: Generate obstacle object element units; S6: Force feedback control for obstacle avoidance by the robotic arm; S7: Object detection and grasping end-effector pose calculation; specifically: Using a front-facing depth camera, a YOLOv convolutional neural network is employed to identify and locate the target, obtaining its position information. Based on this position information, the pose of the robotic arm's end effector is calculated, employing a two-finger robotic arm. The end effector attitude is associated with the position of the grasped target. Assume that the target is located at x, y, z in the coordinate system of the robot arm base. The base coordinate system is x facing forward, y facing to the side, and z facing upward. Using Python, the scipy library is called to calculate the end-effector pose: from scipy.spatial.transform import Rotation as R First, calculate the end effector's orientation with the end face down as the reference orientation for adjustment. At this time, the end effector's coordinate system has the z-axis pointing down, the x-axis pointing to the side, and the y-axis pointing forward. r1=R.from_euler('ZYX',[-3.1415 / 2,3.1415,0]) Using the coordinate system with the end face down as the reference coordinate system, adjust the end face attitude according to the position of the target object; Firstly, to ensure that the two fingers of the two-finger robotic arm face the arm base when the end effector is moving away, and are parallel to the body when moving closer, the end effector rotates around the Z-axis at the following angle: Θ=here(y / x) (x−0.2) (-3) To ensure that the two fingers are tilted outwards when the tip is away from the fuselage and inwards when it is close to the fuselage, the rotation angle of the tip around the X-axis is: Φ=(sqrt(x 2+y 2)-0.4) (-3) To ensure that the end is always parallel to the ground, the end rotates around the Y-axis by 0 degrees. Therefore, the terminal attitude is described as follows: r2=R.from_euler('ZYX',[(Θ,0,Φ]) Finally, the end-effector attitude in the relative base coordinate system is calculated: r=r1 r2； S8: The robotic arm plans the path and performs the grasping action.

2. The method for grasping small objects on the ground with a robotic arm possessing obstacle avoidance capability according to claim 1, characterized in that, Step S1 is as follows: The robot system includes two front-facing depth cameras, namely a left-eye camera and a right-eye camera, a robotic arm hand-eye camera, and a robotic arm. The left-eye camera and the right-eye camera are symmetrically distributed and fixed on the robot. The robotic arm is fixed on the robot via a base. The position of the robot is fixed relative to the left-eye camera and the right-eye camera. The hand-eye camera is fixed at the distal end of the robotic arm's palm joint and moves with the distal end of the robotic arm.

3. The method for grasping small objects on the ground with a robotic arm possessing obstacle avoidance capability according to claim 1, characterized in that, Step S2 is as follows: S21. Point clouds output from two front-facing depth cameras are used as the information source for obstacle avoidance perception; S22. First, uniform downsampling is used to reduce the spacing of the point cloud to 4cm; then, outlier points are filtered out for point clouds with fewer than 6 points within a search radius of 10cm.

4. The method for grasping small objects on the ground with a robotic arm possessing obstacle avoidance capability according to claim 1, characterized in that, Step S4 is as follows: Filter out the ground of the object being grabbed and the surrounding area, including: generating a rectangle with a front-to-back length of 30cm, a left-to-right length of 30cm, and a top-to-bottom length of 8cm, centered on the stone, and removing the collision voxel units within the rectangle.

5. A method for grasping small objects on the ground using a robotic arm with obstacle avoidance capability according to claim 1, characterized in that, Step S5 is as follows: Converting point cloud data into voxel representation involves the following steps: Determine the voxel size: First, it is necessary to determine the size of the voxel, that is, the side length of each voxel; the choice of side length depends on the application requirements and the resolution of the point cloud data. Create a voxel mesh: Create a voxel mesh in 3D space based on the determined voxel size; A voxel grid is a three-dimensional array composed of several voxel units; Mapping point cloud data to a voxel grid: Traverse each point in the point cloud and map it to the corresponding voxel cell according to the point's position; The nearest neighbor method or interpolation method is used to determine the position of the point in the voxel.

6. A method for grasping small objects on the ground using a robotic arm with obstacle avoidance capability according to claim 1, characterized in that, Step S6 is as follows: By using joint force feedback of the robotic arm, when the robotic arm collides with an obstacle, the current trajectory is terminated in time, the robotic arm returns to the previous position, and the robotic arm path is replanned. The method for detecting collisions is to monitor the force feedback of each joint during a normal sampling process over a period of time. Using this range as a benchmark, a bending moment of 2 N / m is added in both positive and negative directions to define the normal force feedback range of the robotic arm joints. If the force feedback exceeds this range, the movement of the robotic arm is terminated.

7. A method for grasping small objects on the ground using a robotic arm with obstacle avoidance capability according to claim 1, characterized in that, S8 specifically refers to: MoveIt is used as the robotic arm planner. By setting the target pose, the target pose of the robotic arm during grasping is determined. Using the planner provided by MoveIt, the target pose is used as input to generate the path of the robotic arm. Finally, the generated path is sent to the robotic arm controller to execute the grasping action. The path execution can be implemented using the interface provided by MoveIt or by writing custom control logic.

8. A robotic arm system for grasping small objects on the ground with obstacle avoidance capability, characterized in that, include: The environment modeling module is used to deploy sensors and collect point clouds for environment modeling. The noise removal module is used to remove noise from the point cloud. The module for generating the imaginary volume of the occluded portion of an obstacle is used to generate the imaginary volume of the occluded portion of an obstacle on the point cloud. The target surrounding obstacle filtering module is used to filter out obstacles around the target on the point cloud; The obstacle object pixel unit generation module is used to generate obstacle object pixels; specifically, it first captures all point clouds above 5cm on the ground, filters out outliers to obtain point clouds of obstacle edges, and then translates the edges backward to generate imaginary volumes. Force feedback control module, used for force feedback control of the robotic arm to avoid obstacles; The object detection and grasping end-effector pose calculation module is used for object detection and grasping end-effector pose calculation; specifically: Using a front-facing depth camera, a YOLOv convolutional neural network is employed to identify and locate the target, obtaining its position information. Based on this position information, the pose of the robotic arm's end effector is calculated, employing a two-finger robotic arm. The end effector attitude is associated with the position of the grasped target. Assume that the target is located at x, y, z in the coordinate system of the robot arm base. The base coordinate system is x facing forward, y facing to the side, and z facing upward. Using Python, the scipy library is called to calculate the end-effector pose: from scipy.spatial.transform import Rotation as R First, calculate the end effector's orientation with the end face down as the reference orientation for adjustment. At this time, the end effector's coordinate system has the z-axis pointing down, the x-axis pointing to the side, and the y-axis pointing forward. r1=R.from_euler('ZYX',[-3.1415 / 2,3.1415,0]) Using the coordinate system with the end face down as the reference coordinate system, adjust the end face attitude according to the position of the target object; Firstly, to ensure that the two fingers of the two-finger robotic arm face the arm base when the end effector is moving away, and are parallel to the body when moving closer, the end effector rotates around the Z-axis at the following angle: Θ=here(y / x) (x−0.2) (-3) To ensure that the two fingers are tilted outwards when the tip is away from the fuselage and inwards when it is close to the fuselage, the rotation angle of the tip around the X-axis is: Φ=(sqrt(x 2+y 2)-0.4) (-3) To ensure that the end is always parallel to the ground, the end rotates around the Y-axis by 0 degrees. Therefore, the terminal attitude is described as follows: r2=R.from_euler('ZYX',[(Θ,0,Φ]) Finally, the end-effector attitude in the relative base coordinate system is calculated: r=r1 r2； The robotic arm path planning and grasping execution module is used for the robotic arm to plan its path and perform grasping.

9. A robotic arm ground object grasping device with obstacle avoidance capability, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement a method for grasping small objects on the ground with an obstacle avoidance capability, as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements a method for grasping small objects on the ground with an obstacle avoidance capability, as described in any one of claims 1-7.

Images