Method for a robot arm to grasp a disordered stack of soles

Abstract

The present application belongs to the technical field of mechanical arm grabbing, and relates to a method for grabbing shoe soles in disordered stacking by a mechanical arm. The method for grabbing shoe soles by a mechanical arm analyzes three links of positioning of shoe soles in disordered stacking, attitude estimation of shoe soles and judgment of shoe sole grabbing priority, and uses disordered grabbing by a mechanical arm and visual detection to grab shoe soles one by one to a processing area, overcomes the problem that shoe soles are difficult to be grabbed with the soles upward in the prior art, improves the grabbing success rate under the condition of disordered stacking of shoe soles, replaces manual picking of shoe soles, and achieves the effect of machine replacing people.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of mechanical arm grabbing, and in particular relates to a method for grabbing disordered stacked shoe soles by a mechanical arm. BACKGROUND

[0002] The existing 3D vision algorithm for shoe sole material recognition is generally realized by a shoe sole surface model matching method. The shoe sole surface model matching method is to obtain the CAD model of the shoe sole to be recognized in advance, then discretize the CAD model into point cloud information and generate a surface feature point set, and match the shoe sole surface model of the front surface of the shoe sole to the best matching object in the scene to be recognized. However, in the disordered stacked state of the shoe sole material, it is difficult to ensure that all the shoe soles are facing the camera, resulting in that the point cloud information after three-dimensional reconstruction does not have a complete front surface of the shoe sole, so that the shoe sole surface model matching method is invalid, and the grabbing work of the shoe sole with the sole up is difficult to realize. Some methods use a vibration disc to vibrate the basket to ensure that the shoe sole faces the camera, and these methods are low in efficiency and produce noise pollution. After analyzing the reasons, it is found that the shoe sole material in the disordered stacked state may face the camera in the reverse and side directions. If a method can be designed to grab the shoe sole with the front surface, the shoe sole with the reverse surface, or the shoe sole with the side surface facing the camera, the grabbing success rate of the shoe sole material in the disordered stacked state will be improved, and the noise pollution caused by the vibration disc will be eliminated. SUMMARY

[0003] The application aims to provide a method for grabbing disordered stacked shoe soles by a mechanical arm, so as to overcome the problem that the existing technology is difficult to realize the grabbing work when the shoe sole is up, and the method is characterized by comprising the following steps:

[0004] Step 1: Point cloud information acquisition: a 3D camera is used to shoot the disordered stacked shoe soles, to obtain the point cloud data of the stacked shoe soles and form a three-dimensional object model;

[0005] Step 2: Point cloud information preprocessing: straight-through filtering is used to remove the background point cloud information of the stacked shoe soles, the point cloud information is down-sampled to reduce the calculation amount of the point cloud processing, and the outlying points in the point cloud information are removed through connected domain analysis to obtain the point cloud information of interest;

[0006] Step 3: Sole surface model making and matching: Take the same model and size of left or right sole or left or right sole side as the sole surface model, use a 3D camera to capture the front and back point cloud information of the sole, and generate a sole surface model from the point cloud information of the sole; calculate the surface normal vector, use the normal vector of the sole surface model to match the point cloud information of interest in the three-dimensional object model to obtain the matching target, and project the sole surface model to the pose transformation matrix of the matching target on the point cloud information of interest in the 3D camera coordinate system; and obtain the pose information of the matching target.

[0007] Step 4: Sole instance segmentation: analyze the connected domain by the Euclidean distance clustering method, find the connected domain that meets the point cloud quantity range and the longest side length range of the left or right sole side in the sole surface model as the segmentation instance, calculate the second moment of the segmentation instance to obtain three principal axis directions as the pose information of the grasping segmentation instance, and calculate the centroid of the segmentation instance to obtain the position information of the grasping;

[0008] Step 5: Grasping priority judgment: In the matching of the sole surface model in step 3 and the sole instance segmentation in step 4, multiple grasping targets in the point cloud information of interest can be obtained, and a judgment function is designed to judge the grasping priority of the grasping target.

[0009] Specifically, the pose transformation of the sole surface model projected onto the point cloud information of interest is a translation matrix and a rotation matrix, i.e., the pose information of the matching target in the 3D camera coordinate system;

[0010] Specifically, in step 2, the point cloud information preprocessing includes the following steps:

[0011] Step 2.1: Straight-through filtering: filter out the material frame and other impurities by x, y, z direction straight-through filtering, and retain the disordered stacked sole point cloud information;

[0012] Step 2.2: Down-sampling processing: select a point as a starting point in the point cloud data, and then select the nearest point within a certain distance for sampling, thereby reducing the number of point clouds and reducing the complexity of calculation;

[0013] Step 2.3: Connected domain analysis: use Euclidean distance clustering analysis, when the distance between two points is less than a certain value, it is considered that the two points belong to the same region, and through this way, search for point cloud clusters gathered together in the point cloud information as a connected domain, and the connected domain with too few points is regarded as an outlier and excluded;

[0014] Specifically, in step 3, the sole surface model matching includes the following steps:

[0015] Step 3.1: The sole surface model is created by sampling the 3D object model with a certain distance on the point cloud information, allowing the sole surface model to be scaled down and up by a certain ratio;

[0016] Step 3.2: Feature extraction: Extract the surface point and normal vector features of the point cloud information of interest and the sole surface model obtained in step 1: These features can depict the shape and geometry in the point cloud;

[0017] Step 3.3: Feature matching: Compare the feature similarity between the point cloud information of interest and the sole surface model;

[0018] Step 3.4: Pose estimation: After finding the best feature matching pair, the best feature matching pair is regarded as the matching target of the sole surface model in the point cloud information of interest, and the translation and rotation transformation of the matching target relative to the sole surface model is the pose information of the matching target;

[0019] Step 3.5: Matching evaluation: Evaluate the accuracy of the matching, and the evaluation index is the matching similarity, which is the ratio between the number of corresponding feature points of the feature points of the sole surface model in the point cloud information of interest and the total number of feature points of the sole surface model, the value is between 0 and 1.

[0020] Specifically, in the step 4: the sole instance segmentation includes the following steps:

[0021] Step 4.1: Cluster segmentation: Analyze the connected domain by the method of Euclidean distance clustering, find the connected domain that meets the point cloud number range and the longest side length range of the sole side, as the segmentation instance;

[0022] Step 4.2: Pose calculation: Calculate the second moment of the segmentation instance to obtain three principal axis directions as the pose information of grabbing the instance;

[0023] Step 4.3: Centroid position calculation: Calculate the centroid of the minimum circumscribed rectangle of the segmentation instance to obtain the position information of grabbing the instance.

[0024] Specifically, in the step 5: the grabbing priority judgment includes the following steps:

[0025] Step 5.1: Determine whether there is a matching target: Check whether there is a matching target, if there is a matching target, proceed to step 5.2, if there is no matching target, check whether there is a segmentation instance, if the instance exists, proceed to step 5.5, if the instance does not exist, it means that there is no grabbable sole in the field of view of the 3D camera, at this time the material basket should be replaced;

[0026] Step 5.2: Determine whether there is a split instance: check whether there is a split instance, if there is, go to step 5.3, if not, go to step 5.4;

[0027] Step 5.3: Determine the priority of the matching target and the split instance: if the split instance is above the matching target, first grab the split instance, and go to step 5.5, otherwise, first grab the matching target, and go to step 5.4;

[0028] Step 5.4: Determine the priority of the matching target: if there are multiple matching targets, set a judgment function to determine the priority of the matching target according to the height of the matching target, the matching similarity and other conditions, and the judgment function is:

[0029]

[0030] Where P is the priority, Z is the height in the field of view of the 3D camera, Z offset is the height compensation, score_array is the matching similarity, the value is between 0 and 1, H and S are the weight coefficients of height and similarity respectively, and the specific values are determined by debugging.

[0031] Step 5.5: Determine the priority of the split instance: if there are multiple split instances, preferentially grab the split instance at the highest height.

[0032] Compared with the prior art, the beneficial effects of the present application are: the method for grabbing the shoe sole by the mechanical arm analyzes through the three links of positioning of the disordered stacked shoe sole, attitude estimation of the shoe sole and judgment of the shoe sole grabbing priority, uses the mechanical arm to grab and visual detection to grab the shoe sole one by one to the processing area, overcomes the problem that the prior art is difficult to achieve grabbing work when the shoe sole is upward, improves the grabbing success rate of the shoe sole in the disordered stacked state, replaces manual picking of the shoe sole, and achieves the effect of machine replacing man. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a total step flowchart of the method for grabbing the disordered stacked shoe sole by the mechanical arm of the present application;

[0034] Figure 2 is a step flowchart of the method for grabbing the disordered stacked shoe sole by the mechanical arm of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0036] Referring to Figure 1 Figure 2 A mechanical arm grasps the unordered stacked shoe sole method provided by the application is a two-step flow chart. The mechanical arm grasping method provided by the application includes three links of positioning of unordered stacked shoe soles, attitude estimation of shoe soles and judgment of shoe sole grasping priority, and the purpose is to use a mechanical arm to grasp one by one the shoe soles placed in the material frame in an unordered stack to a processing area, so as to replace manual picking of shoe soles, and achieve the effect of machine replacing people.

[0037] The mechanical arm grasps the unordered stacked shoe sole method, characterized by comprising the following steps:

[0038] (1) Point cloud information acquisition: using a 3D camera to shoot the unordered stacked shoe soles, obtaining the stacked shoe sole point cloud data, and forming a three-dimensional object model.

[0039] (2) Point cloud information preprocessing: using straight-through filtering to remove background point cloud information from the stacked shoe sole point cloud data, performing down-sampling processing on the point cloud information to reduce the calculation amount of point cloud processing, and removing outliers in the point cloud information through connected domain analysis to obtain the point cloud information of interest.

[0040] (3) Shoe sole surface model making and matching: taking out a left shoe sole or a right shoe sole or a left shoe sole side or a right shoe sole side of the same model and the same size as the shoe sole surface model to be grasped, using a 3D camera to shoot the front and back point cloud information of the shoe sole, generating a shoe sole surface model from the shoe sole point cloud information; calculating the surface normal vector, using the normal vector of the shoe sole surface model to match the point cloud information of interest in the three-dimensional object model to obtain a matching target, and projecting the shoe sole surface model to the pose transformation matrix (translation matrix and rotation matrix) of the point cloud information of interest on the pose transformation matrix (translation matrix and rotation matrix) in the 3D camera coordinate system, i.e. the pose information of the matching target, for positioning of the unordered stacked shoe soles.

[0041] (4) Shoe sole instance segmentation: analyzing the connected domain by the Euclidean distance clustering method, finding the connected domain that meets the point cloud quantity range and the longest side length range of the left shoe sole side or the right shoe sole side in the shoe sole surface model as the segmentation instance, calculating the second moment of the segmentation instance to obtain three principal axis directions as the attitude information of the grasped segmentation instance, calculating the centroid of the segmentation instance to obtain the position information of the grasped segmentation instance, and estimating the attitude of the shoe sole.

[0042] (5) Grasping priority judgment: In step (3) the sole surface model matching and step (4) the sole instance segmentation can obtain multiple grasping targets in the point cloud information of interest, a judgment function is designed to judge the grasping priority of the grasping target, and the grasping priority of the sole is judged. Through the above selection of priority, different sole states are identified one by one, the mechanical arm successfully grasps the sole, and the problem that the prior art is difficult to achieve grasping work when the sole is upward is overcome, and the grasping success rate under the disordered stacking state of the sole is improved.

[0043] The above step (2) point cloud information preprocessing includes the following steps:

[0044] (2.1) Straight-through filtering: Through straight-through filtering in x, y and z directions, impurities such as material frames are filtered out, and point cloud information of disordered stacked soles is retained.

[0045] (2.2) Down-sampling processing: Through distance-based sampling, a point in the point cloud data is selected as a starting point, and then the nearest point within a certain distance is selected for sampling, thereby reducing the number of point clouds and reducing the complexity of calculation.

[0046] (2.3) Connected domain analysis: Euclidean distance clustering analysis is used, when the distance between two points is less than a certain value, it is considered that the two points belong to the same region, and through this way, the point cloud clusters gathered together in the point cloud information are searched as a connected domain, and the connected domain with too few points is regarded as an outlier and excluded.

[0047] The above step (3) sole surface model matching includes the following steps:

[0048] (3.1) The sole surface model is created by sampling a three-dimensional object model on the point cloud information at a certain distance, allowing the sole surface model to be scaled down and up by a certain ratio, so that one sole surface model can match soles of multiple sizes.

[0049] (3.2) Feature extraction: The surface points and normal vector features of the point cloud information of interest obtained in step (2) and the sole surface model obtained in step (1) are extracted, which can depict the shape and geometric structure in the point cloud.

[0050] (3.3) Feature matching: Compare the feature similarity between the point cloud information of interest and the sole surface model.

[0051] (3.4) Pose estimation: After finding the best feature matching pair, the best feature matching pair is regarded as the matching target of the sole surface model in the point cloud information of interest, and the translation and rotation transformation of the matching target relative to the sole surface model is the pose information of the matching target.

[0052] (3.5) Matching evaluation: Evaluate the accuracy of the matching, the evaluation index is the matching similarity, which is the ratio between the number of feature points of the sole surface model that have corresponding feature points in the point cloud information of interest and the total number of feature points of the sole surface model, the value is between 0 and 1.

[0053] In the above step (4) of sole instance segmentation, the following steps are included:

[0054] (4.1) Cluster segmentation: Analyze the connected domains by the method of Euclidean distance clustering, and find the connected domains that meet the point cloud quantity range and the longest side length range of the sole side surface as the segmented instances.

[0055] (4.2) Pose calculation: Obtain the three principal axis directions by calculating the second moment of the segmented instance as the pose information for grasping the instance.

[0056] (4.3) Centroid position calculation: Obtain the position information for grasping the instance by calculating the centroid of the minimum circumscribed rectangle of the segmented instance.

[0057] In the above step (5) of grasping priority judgment, the following steps are included:

[0058] (5.1) Judgment of whether there is a matching target: Check whether there is a matching target, if there is a matching target, proceed to step (5.2), if there is no matching target, check whether there is a segmented instance, if the instance exists, proceed to step (5.5), if the instance does not exist, it means that there is no graspable sole in the field of view of the 3D camera, at this time the material basket should be replaced.

[0059] (5.2) Judgment of whether there is a segmented instance: Check whether there is a segmented instance, if there is, proceed to step (5.3), if there is not, proceed to step (5.4).

[0060] (5.3) Judgment of the priority of the matching target and the segmented instance: If the segmented instance is above the matching target, the segmented instance is grasped first, and step (5.5) is performed, otherwise the matching target is grasped first, and step (5.4) is performed.

[0061] (5.4) Judgment of the priority of the matching target: If there are multiple matching targets, a discrimination function is set to judge the priority of grasping in combination with the height of the matching target, the matching similarity and other conditions, and the discrimination function is

[0062]

[0063] Where P is the priority, Z is the height in the field of view of the 3D camera, Z offset is the height compensation, which is to ensure The priority of the grabbing is proportional to the height of the matching target, the higher the height of the matching target, the higher the priority of the grabbing, score array is the matching similarity, the value is between 0 and 1, H and S are weight coefficients of the height and the similarity respectively, and specific values are determined through debugging.

[0064] (5.5) judging the priority of the segmented instance: if there are multiple segmented instances, the segmented instance at the highest position is preferentially grabbed.

[0065] The above is only one of the preferred embodiments of the present application, and does not limit the present application, and does not limit the patent implementation scope of the present application, and any equivalent structure or equivalent process transformation using the shape, structure, principle of the content of the specification and drawings, or directly or indirectly used in other related technical fields; any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application, are covered by the patent protection scope of the present application.

Claims

1. A method of a robotic arm grabbing a disordered stack of shoe soles, characterized by, The method comprises the following steps: Step 1: Point cloud information collection: using a 3D camera to shoot the disordered stacked soles, obtaining the point cloud data of the stacked soles, and forming a three-dimensional object model; Step 2: Point cloud information preprocessing: using straight-through filtering to remove background point cloud information from the point cloud data of the stacked soles, performing down-sampling processing on the point cloud information to reduce the calculation amount of the point cloud processing, and removing outliers in the point cloud information through connected domain analysis to obtain the point cloud information of interest; Step 3: Sole surface model making and matching: taking out a left sole or a right sole or a left sole side or a right sole side of the same model and the same size as the sole surface model to be grabbed, using a 3D camera to shoot the front and back point cloud information of the sole, generating a sole surface model from the point cloud information of the sole, calculating the surface normal vector, using the normal vector of the sole surface model to match the point cloud information of interest in the three-dimensional object model to obtain a matching target, and projecting the sole surface model to the pose transformation matrix of the matching target on the point cloud information of interest in the 3D camera coordinate system to obtain the pose information of the matching target; Step 4: Sole instance segmentation: analyzing the connected domain through the Euclidean distance clustering method, finding the connected domain that meets the point cloud quantity range and the longest side length range of the left sole side or the right sole side in the sole surface model as the segmentation instance, calculating the second moment of the segmentation instance to obtain three principal axis directions as the pose information of the segmentation instance, and calculating the centroid of the segmentation instance to obtain the position information of the segmentation instance; Step 5: Judgment of the priority of grabbing: multiple grabbing targets in the point cloud information of interest can be obtained through the matching of the sole surface model in step 3 and the segmentation of the sole instance in step 4, and a judgment function is designed to judge the priority of grabbing the targets; In step 5: the judgment of the priority of grabbing comprises the following steps: Step 5.1: judging whether there is a matching target: checking whether there is a matching target, if there is a matching target, proceeding to step 5.2, if there is no matching target, checking whether there is a segmentation instance, if the instance exists, proceeding to step 5.5, and if the instance does not exist, it means that there is no sole that can be grabbed in the field of view of the 3D camera, and at this time, the material basket should be replaced; Step 5.2: judging whether there is a segmentation instance: checking whether there is a segmentation instance, if there is, proceeding to step 5.3, if there is not, proceeding to step 5.4; Step 5.3: judging the priority of the matching target and the segmentation instance: if the segmentation instance is above the matching target, the segmentation instance is grabbed first, and step 5.5 is performed, otherwise, the matching target is grabbed first, and step 5.4 is performed; Step 5.4: judging the priority of the matching target: if there are multiple matching targets, a judgment function is set to judge the priority of grabbing by combining the height of the matching target, the matching similarity and other conditions, and the judgment function is: ； wherein P is the priority, Z is the height in the 3D camera view, Z offset is the compensation amount of height, score_array is the matching similarity, value between 0 and 1, H and S are the weight coefficients of height and similarity respectively, the specific value is determined by debugging. Step 5.5: judging the priority of the segmentation instance: if there are multiple segmentation instances, the segmentation instance at the highest position is grabbed first.

2. The method of claim 1, wherein, The posture transformation of the shoe sole surface model projected onto the point cloud information of interest through rigid body transformation is a translation matrix and a rotation matrix, that is, the pose information of the matching target in the 3D camera coordinate system.

3. The method of claim 1, wherein, In the step 2, the point cloud information preprocessing includes the following steps: Step 2.1: straight-through filtering: through x, y, z direction straight-through filtering, filtering out sundries such as material frame, and retaining disordered stacked shoe sole point cloud information; Step 2.2: downsampling processing: through distance-based sampling, selecting a point as the starting point in the point cloud data, and then selecting the nearest point within a certain distance for sampling, thereby reducing the number of point clouds and reducing the complexity of calculation; Step 2.3: connected domain analysis: using Euclidean distance clustering analysis, when the distance between two points is less than a certain value, it is considered that the two points belong to the same region, and through this way, the point cloud clusters gathered together in the point cloud information are searched as a connected domain, and the connected domain with too few points is regarded as an outlier and excluded.

4. The method of claim 1, wherein, In the step 3, the shoe sole surface model matching includes the following steps: Step 3.1: the shoe sole surface model is created by sampling the three-dimensional object model on the point cloud information with a certain distance, allowing the shoe sole surface model to be scaled down and up by a certain ratio; Step 3.2: feature extraction: extracting surface points and point normal vector features from the point cloud information of interest obtained in step 2 and the shoe sole surface model obtained in step 1, which can depict the shape and geometric structure in the point cloud; Step 3.3: feature matching: comparing the feature similarity between the point cloud information of interest and the shoe sole surface model; Step 3.4: pose estimation: after finding the best feature matching pair, the best feature matching pair is regarded as the matching target of the shoe sole surface model in the point cloud information of interest, and the translation and rotation transformation of the matching target relative to the shoe sole surface model is the pose information of the matching target; Step 3.5: matching evaluation: evaluating the accuracy of the matching, and the evaluation index is the matching similarity, that is, the ratio between the number of corresponding feature points of the feature points of the shoe sole surface model in the point cloud information of interest and the total number of feature points of the shoe sole surface model, the value is between 0 and 1.

5. The method of claim 1, wherein, In the step 4, the shoe sole instance segmentation includes the following steps: Step 4.1: cluster segmentation: analyzing the connected domain through the method of Euclidean distance clustering, and finding the connected domain that meets the range of point cloud number and the length of the longest side of the shoe sole side as the segmentation instance; Step 4.2: pose calculation: obtaining three principal axis directions by calculating the second moment of the segmentation instance as the pose information of grasping the instance; Step 4.3: centroid position calculation: obtaining the position information of grasping the instance by calculating the centroid of the minimum circumscribed rectangle of the segmentation instance.

Images