Random grabbing method for mechanical arm

Abstract

The application provides a mechanical arm random grabbing method, which greatly reduces the beat time of the grabbing system by detecting the changed target object in the scene and only estimating the pose of the changed target object. By tracking and establishing a target database, the global target object pose is updated after each grabbing, which has no influence on the performance of the subsequent motion estimation module, and the globally optimal target object can be selected for grabbing. By designing a new change area target detection method, only the changed object can be detected, and the robustness is high, and the interference caused by the light environment, slight jitter and background change is excluded. The application detects the changed target object in the scene, only estimates the pose of the changed target object, and has high grabbing efficiency and shortened beat time. Instead of performing pixel-level difference operation, a new change area target detection method is used to directly output the changed target object, and the changed target object detection is accurate and has high robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of machine vision and motion control. BACKGROUND

[0002] The most core component in the mechanized automation of production manufacturing is the mechanical arm, which is the most important part of the replacement of human resources. An important problem that the mechanical arm needs to solve is to complete the grasping and manipulation of various materials under the guidance of vision. In simple terms, it simulates the eyes and arms of workers to cooperatively complete the feeding and placement of materials. This problem is currently a hot topic in the fields of machine vision and robotics, which involves multiple complex topics such as 3D image segmentation, 3D object recognition, 6D pose estimation, motion planning, and mechanical arm control.

[0003] In the existing scheme, the first step is to collect 2D or 3D depth images through sensors, and then input them into an estimation module to estimate the spatial pose of each target object. After obtaining different poses, the subsequent algorithm module selects an easy-to-grasp point according to the pre-defined grasping point. The subsequent motion planning algorithm calculates an optimal motion trajectory based on the determined grasping point to control the robot to complete the grasping of the target object and move it to the specified position. After the overall grasping is completed, it returns to the first step for the next round of grasping. According to the relative position relationship between the sensor and the mechanical arm, the system is divided into two deployment modes:

[0004] 1) Fixed perspective (eye-to-hand) deployment: In this mode, the sensor is in a fixed position in the entire system, and the scene data collection is completed when the target area is exposed to the sensor field of view.

[0005] 2) Arm perspective (eye-in-hand) deployment: In this mode, the sensor is located above the mechanical arm, and the scene data collection is completed when the mechanical arm moves over the target area.

[0006] Regardless of the deployment scheme, the most important performance indicator of a grasping system is the cycle time, which is the time interval between each time a part is grasped and placed in the specified position. To optimize the cycle time, the traditional scheme often takes the following measures:

[0007] 1) Asynchronous data capture: After the mechanical arm grasps the target object, the sensor starts data collection immediately after the target area is moved away without obstruction.

[0008] 2) Limit the maximum number of object detection: The system limits the number of detected objects to shorten the total detection time of the entire scene.

[0009] 3) Detect the changed area: Use pixel difference to detect the changed area and re-detect the objects in the changed area.

[0010] Although the current beat time optimization method shortens the beat time to a certain extent, it also brings many shortcomings and limitations, as follows:

[0011] 1) Although the use of asynchronous data capture can speed up the beat time, it still cannot avoid the constraints of linear flow. That is, only after the sensor captures the scene and completes the pose estimation of the target object, can the grabbing start. In actual scenarios, there are often a large number of target objects, and the pose estimation module at this time will become the performance bottleneck of the entire grabbing system, causing the beat time to increase linearly with the number of target objects.

[0012] 2) In order to alleviate the increase of beat time caused by a large number of target objects, it is necessary to limit the maximum number of object detection. However, this will bring another problem: since the pose estimation module does not estimate the pose of all target objects in the scene, the subsequent motion planning cannot obtain a globally optimal grabbing target and optimal path planning scheme, but can only select a local optimal solution from the limited targets.

[0013] 3) The pixel difference algorithm for detecting the changed area has low robustness, is sensitive to environmental light and jitter, and cannot distinguish between actual object changes and background changes. The problems of missed detection or repeated calculation may occur. SUMMARY

[0014] The technical problem to be solved by the present application is to provide a more efficient random grabbing method of a mechanical arm.

[0015] The technical solution adopted by the present application to solve the above technical problem is a random grabbing method of a mechanical arm, comprising the following steps:

[0016] 1) After the system is powered on, initialize all peripherals and parameters, and empty the target database;

[0017] 2) The sensor module starts to image the scene in 2D or 3D, and sends the scene data to the detection and tracking module;

[0018] 3) The detection and tracking module detects the target object area changed after the last clamping in the scene according to the scene data combined with the existing results in the target database through comprehensive judgment, and outputs the target object area to the pose estimation module; all target objects detected in the initial frame scene data are changed target objects;

[0019] 4) The pose estimation module identifies the target pose of the changed recognition change region according to the changed target object region input by the detection tracking module, and updates the changed target pose recognition result to the target database, wherein the target database stores the pose recognition results of all target objects to be grabbed in the current scene; the target database saves the global information of all objects with changed poses and objects without changed poses, and a subsequent module can select an optimal solution according to the global information;

[0020] 5) The motion planning module retrieves the poses of all target objects to be grabbed from the target database, and determines an optimal clamping target and clamping point according to a preset rule, so as to calculate the motion trajectory and operation instruction list used for clamping;

[0021] 6) The robot arm control module controls the robot arm according to the motion trajectory and operation instruction list, and completes the grabbing and placing of the target object;

[0022] 7) After the target object is grabbed, the verification and update module verifies the scene to determine whether the grabbing is successful, and if yes, the target database is updated to delete the grabbed target object, and if not, the information in the target database remains unchanged;

[0023] 8) The system determines whether the clamping of all target objects in the scene is completed, and if not, returns to step 2).

[0024] The present application greatly reduces the tact time of the grabbing system by detecting the changed target object in the scene and only estimating the pose of the changed target object. By tracking and establishing the target database, the global target object pose is updated after each grabbing, and the performance of the subsequent motion estimation module is not affected, and the globally optimal target object can be selected for grabbing. By using a new change region target detection method, only the changed object can be detected, and the robustness is high, and the interference caused by the light environment, slight jitter and background change is excluded.

[0025] The present application has the advantages that the changed target object in the scene is detected, and only the pose of the changed target object is estimated, so that the grabbing efficiency is high and the tact time is shortened. Instead of performing pixel-level difference operation, a new change region target detection method is used to directly output the changed target object, so that the changed target object detection is accurate and the robustness is high. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The method flowchart is for an embodiment.

[0027] Figure 2 The working schematic block diagram of the detection tracking module is shown.

[0028] Figure 3Fig. 1 is a schematic diagram of a changed region target detection network.

[0029] Figure 4 Fig. 2 is a schematic diagram of a changed region recommendation network. DETAILED DESCRIPTION

[0030] The embodiment of the system includes a sensor module, a detection tracking module, a target database, a pose estimation module, a motion planning module, a robot arm control module, and a verification and update module.

[0031] The implementation process of the system during the grabbing is shown in Fig. 3. Figure 1

[0032] 1) After the system is powered on, all peripherals and parameters are initialized, and the target database is emptied.

[0033] 2) The sensor module starts to image the scene in 2D or 3D, and sends the scene data to the detection tracking module.

[0034] 3) The detection tracking module detects the changed target object region in the scene after the last clamping according to the scene data combined with the existing results in the target database through comprehensive judgment, and outputs the target object region to the pose estimation module. All the detected target objects in the initial frame scene data are changed target objects.

[0035] 4) The pose estimation module identifies the pose of the changed target object region according to the changed target object region input by the detection tracking module, updates the changed target pose recognition result to the target database, and stores the pose recognition result of all the target objects to be grabbed in the current scene in the target database. The target database saves the global information of all the objects with changed poses and objects without changes, and the subsequent modules can select an optimal solution according to the global information.

[0036] 5) The motion planning module retrieves the poses of all the target objects to be grabbed from the target database, determines an optimal clamping target and clamping point according to the preset rules, and calculates the motion trajectory and operation instructions to obtain the motion trajectory and operation instruction list.

[0037] 6) The robot arm control module controls the robot arm according to the motion trajectory and operation instruction list to complete the grabbing and placing of the target object.

[0038] 7) After the target object is grabbed, the verification and update module verifies the scene to determine whether the grabbing is successful. If yes, the target database is updated to delete the grabbed target object. If no, the information in the target database remains unchanged.

[0039] ​8) The system determines whether the grasping of all target objects in the scene is completed. If not, return to step 2).

[0040] The core of the whole system is the design of the detection and tracking module, which is a new change area target detection network for identifying the target objects in the scene that change in pose each time. Thus, the subsequent pose estimation module only needs to process the target objects that change in pose, thereby speeding up the beat time of the overall system.

[0041] The detection and tracking module includes a data preprocessing module, a change area target detection network, and a target tracking algorithm processing module. The workflow is as shown in Figure 2

[0042] 1) The data preprocessing module fuses the current scene data S and the historical scene data S' and inputs them into the change area target detection network. The historical scene data is the scene data received by the data preprocessing module before;

[0043] 2) The change area target detection network detects the target objects that change and outputs the changed target list O to the target tracking algorithm processing module. The changed target list is represented using the bounding box of the target;

[0044] 3) The target tracking algorithm processing module compares the records E in the target database with the changed target list O, excludes the small disturbance objects in the changed target list O, obtains the changed target boundary set O', and outputs the changed target boundary set O' to the pose estimation module;

[0045] 4) The pose estimation module estimates the pose of the changed target object according to the changed target boundary set O' and updates the pose records in the target database.

[0046] The change area target detection network includes a convolutional neural network CNN, a region of interest pooling layer ROIPooling, a change area recommendation network, and a prediction head. The workflow is as shown in Figure 3

[0047] 1) The data preprocessing module fuses the w x h x c current scene data S and the w x h x c historical scene data S' through a scene data S' fusion module and inputs the fused data into the change area target detection network. The dimension of the fused data is w x h x 2c;

[0048] 2) The fused data is processed by a convolutional neural network CNN in the change area target detection network to extract feature vectors and form a feature map;

[0049] 3) The change area recommendation network predicts the position and size of the changed object according to the feature map; ​​

[0050] 4) Interest region pooling layer performs pooling operation on the original feature vector map and outputs the corresponding feature vector;

[0051] 5) The prediction head composed of multiple fully connected layers finally regresses the feature vector into the predicted change object bounding box, i.e. the object list O that changes.

[0052] The change region recommendation network includes a convolutional neural network CNN, an anchor box generation module, a prediction module, and a non-maximum suppression processing module. The workflow is as shown in Figure 4

[0053] 1) The convolutional neural network CNN in the change region recommendation network outputs an MN-dimensional feature map according to the input of the global feature map, which is used to predict whether there is an object change in the receptive field represented by each grid; for example, the actual change object region is the gray area in Figure 4 , then the value of the changed part in the prediction vector remains unchanged, i.e. the grid with value x in the figure, otherwise it outputs zero;

[0054] 2) After receiving the input feature map, the anchor box generation module generates anchor boxes according to different scales and sizes defined in advance;

[0055] 3) The prediction module performs binary classification prediction and bounding box prediction on all generated anchor box regions to determine whether there is a target object in the anchor box to determine the bounding box of the target object;

[0056] 4) The non-maximum suppression processing module performs non-maximum suppression on the bounding box of the predicted target object to obtain the final result, thereby outputting the position and size of the predicted changed object.​

Claims

1. A method for random grasping by a robotic arm, characterized in that, Includes the following steps: 1) After the system powers on, it initializes all peripherals and parameters and clears the target database; 2) The sensor module begins to perform 2D or 3D imaging of the scene and sends the scene data to the detection and tracking module; 3) The detection and tracking module, based on scene data and existing results in the target database, makes a comprehensive judgment to detect the target object region in the scene that has changed since the last gripping, and outputs the target object region to the pose estimation module. In the initial frame scene data input, all detected target objects are target objects that have undergone changes; 4) The pose estimation module identifies the target pose of the changed target object region input by the detection and tracking module, and updates the target pose identification results to the target database. The target database stores the pose identification results of all target objects to be captured in the current scene. The target database also stores global information of all objects with changed poses and those without. Subsequent modules can select an optimal solution based on the global information. 5) The motion planning module retrieves the poses of all the target objects to be grasped from the target database, and then determines an optimal grasping target and grasping point according to preset rules, thereby calculating the motion trajectory and operation instructions used to complete the grasping and obtaining a list of motion trajectory and operation instructions. 6) The robotic arm control module controls the robotic arm according to the motion trajectory and operation instruction list to complete the grasping and placement of the target object; 7) After the target object is captured, the verification and update module verifies the scene to determine whether the capture was successful. If successful, the captured target object is deleted from the target database. If not, the information in the target database remains unchanged. 8) The system determines whether the grabbing of all target objects in the scene has been completed; otherwise, it returns to step 2.

2. The method as described in claim 1, characterized in that, The detection and tracking module includes a data preprocessing module, a target detection network for changing regions, and a target tracking algorithm processing module. The specific workflow of the detection and tracking module is as follows: The data preprocessing module fuses the current scene data S and the historical scene data S′ and feeds them into the change area target detection network; the historical scene data is the scene data previously received by the data preprocessing module. The changing region target detection network detects target objects that have changed and outputs a list of changed targets to the target tracking algorithm processing module; the list of changed targets is represented by the bounding box of the target. The target tracking algorithm processing module compares the record E in the target database with the list of changed targets O. After excluding objects with minor perturbations in the target list O, it obtains the set of changed target boundaries O′. The set of changed target boundaries O′ is then output to the pose estimation module as the target object region that has changed since the last gripping in the scene.

3. The method as described in claim 2, characterized in that, The changing region object detection network includes a convolutional neural network (CNN), a region of interest pooling layer, a changing region recommendation network, and a prediction head. The specific workflow of the changing region object detection network is as follows: 1) The data preprocessing module fuses the current scene data S (w×h×c) and the historical scene data (w×h×c) through the scene data S′ fusion module and inputs the data into the change area target detection network. The fused data dimension is w×h×2c. 2) The fused data is processed by a convolutional neural network (CNN) in the change region target detection network to extract feature vectors and form feature maps; 3) The change region recommendation network predicts the location and size of objects that have changed based on feature maps; 4) The region of interest pooling layer performs pooling operations on the original feature vector map and outputs the corresponding feature vector; 5) Finally, the prediction head, composed of multiple fully connected layers, regresses the feature vectors into a prediction of the changed target list O.

4. The method as described in claim 3, characterized in that, The changing region recommendation network includes a convolutional neural network (CNN), an anchor box generation module, a prediction module, and a non-maximum suppression module. The specific workflow of the changing region recommendation network is as follows: 1) The Convolutional Neural Network (CNN) in the Change Region Recommendation Network outputs a feature map based on the input feature map, which predicts whether there is a change in the object within the receptive field represented by each square. 2) After receiving the input feature map, the anchor box generation module generates anchor boxes according to different predefined proportions and sizes; 3) The prediction module performs binary classification prediction and bounding box prediction on all generated anchor box regions to determine whether there is a target object within the anchor box and then determines the bounding box of the target object. 4) The non-maximum suppression processing module performs non-maximum suppression on the bounding box of the predicted target object to obtain the final result, thereby outputting the predicted position and size of the object that has changed.

Images