Grasping method, device and computing equipment for objects with pure planar structure

Abstract

The present invention discloses a method, device, and computing device for grasping objects with purely planar structures, wherein the method comprises: obtaining point clouds corresponding to multiple objects in the current scene; for each object, performing edge extraction on the point cloud corresponding to the object to obtain the edge point cloud corresponding to the object, and calculating the tangent vectors of each 3D point in the edge point cloud; for any two 3D points in the edge point cloud, constructing a point pair containing the two 3D points, and generating a point pair feature vector of the point pair based on the tangent vectors of the two 3D points; matching the edge point cloud corresponding to the object with a preset template point cloud based on the point pair feature vectors of each point pair of the object to obtain the pose information of the object. This solution generates point pair feature vectors of the point pair based on the tangent vectors of the 3D points in the edge point cloud, and matches the edge point cloud with the preset template point cloud based on the point pair feature vectors, thereby achieving rapid and accurate recognition of the pose information of the object.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device and computing equipment for grasping objects with pure planar structures. Background Art

[0002] With the development of industrial intelligence, the use of robots to manipulate objects (such as industrial parts and boxes) instead of humans is becoming increasingly common. Robotic operation typically involves grasping objects, moving them from one location, and placing them in another. For example, robots can grab objects from a conveyor belt, move them, and place them on a pallet or in a cage. Alternatively, robots can grab objects from a pallet and place them on a conveyor belt or other pallet as required. However, existing technologies for recognizing the positional information of objects to be grasped lack accuracy and efficiency, making it difficult to meet the demands of high-speed industrial automation. Summary of the Invention

[0003] In view of the above problems, the present invention is proposed to provide a method, apparatus and computing device for grasping objects with purely planar structures that overcome the above problems or at least partially solve the above problems.

[0004] According to one aspect of the present invention, a method for grasping an object with a purely planar structure is provided, the method comprising:

[0005] Obtaining point clouds corresponding to multiple objects in the current scene; wherein the multiple objects have a purely planar structure;

[0006] For each object, perform edge extraction on the point cloud corresponding to the object to obtain the edge point cloud corresponding to the object, and calculate the tangent vector of each 3D point in the edge point cloud;

[0007] For any two 3D points in the edge point cloud, construct a point pair containing the two 3D points, and generate the point pair feature vector based on the tangent vector of the two 3D points;

[0008] According to the point pair feature vectors of each point pair of the object, the edge point cloud corresponding to the object is matched with the preset template point cloud to obtain the posture information of the object so that the robot can perform a grasping operation based on the posture information of the object.

[0009] According to another aspect of the present invention, there is provided a gripping device for an object having a purely planar structure, the device comprising:

[0010] A first acquisition module is adapted to acquire point clouds corresponding to a plurality of objects in a current scene; wherein the plurality of objects have a purely planar structure;

[0011] The edge extraction module is suitable for performing edge extraction on the point cloud corresponding to each object, obtaining the edge point cloud corresponding to the object, and calculating the tangent vector of each 3D point in the edge point cloud;

[0012] The point pair construction module is suitable for constructing a point pair containing two 3D points for any two 3D points in the edge point cloud, and generating a point pair feature vector based on the tangent vectors of the two 3D points;

[0013] The matching module is suitable for matching the edge point cloud corresponding to the object with the preset template point cloud based on the point pair feature vectors of each point pair of the object to obtain the posture information of the object so that the robot can perform a grasping operation based on the posture information of the object.

[0014] According to another aspect of the present invention, there is provided a computing device comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0015] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned method for grasping objects with pure planar structures.

[0016] According to another aspect of the present invention, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to execute operations corresponding to the above-mentioned method for grasping objects with a purely planar structure.

[0017] According to the technical solution provided by the present invention, the pure planar structural features are fully studied, the edge point cloud is extracted from the point cloud corresponding to the object, and the point pair feature vectors of the point pairs are generated based on the tangent vectors of the 3D points in the edge point cloud. The shape and structural features of the object are accurately reflected through the point pair feature vectors; the edge point cloud is matched with the preset template point cloud according to the point pair feature vector, so as to realize the accurate recognition of the posture information of the object, which helps the robot to accurately and firmly perform the grasping operation according to the posture information of the object, and avoids grasping errors such as the robot being unable to successfully grasp the object or the object falling after being grasped; and, in this solution, only the edge point cloud corresponding to the object is extracted to participate in the matching process, and the non-edge point cloud does not participate in the matching process, thereby effectively reducing the matching workload and improving the recognition efficiency of the posture information of the object.

[0018] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0020] Figure 1 A schematic flow chart of a method for grasping an object with a purely planar structure according to an embodiment of the present invention is shown;

[0021] Figure 2 A schematic flow chart of a method for grasping an object with a purely planar structure according to another embodiment of the present invention is shown;

[0022] Figure 3 A structural block diagram of a gripping device for an object with a purely planar structure according to an embodiment of the present invention is shown;

[0023] Figure 4 A schematic structural diagram of a computing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0024] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0025] Figure 1 FIG. 1 shows a flow chart of a method for grasping an object with a purely planar structure according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0026] Step S101: Obtain point clouds corresponding to multiple objects in the current scene.

[0027] The current scene contains multiple objects, many of which have purely planar structures, such as rectangular boxes. In step S101, a pre-processed point cloud corresponding to the multiple objects in the current scene is obtained. The point cloud includes the pose information of each 3D point, which may specifically include the X, Y, and Z coordinates of each 3D point in space and the X, Y, and Z directions of each 3D point. After obtaining the point clouds corresponding to the multiple objects, each object in the current scene is processed sequentially according to steps S102 through S104.

[0028] In step S102 , for each object, edge extraction is performed on the point cloud corresponding to the object to obtain the edge point cloud corresponding to the object, and the tangent vector of each 3D point in the edge point cloud is calculated.

[0029] Among them, for each object, the edge point cloud corresponding to the object can be extracted from the point cloud corresponding to the object based on 3D or 2D projection and other methods. During the invention process, the inventor carefully analyzed the edge point cloud corresponding to the object with a purely planar structure and learned that when objects of different shapes are in a flat state, the normal vector characteristics of each 3D point in its edge point cloud are the same, and the normal direction is vertically upward, but its tangent vector is different, so the tangent vector can be used to reflect the shape and structural characteristics of the object. After extracting the edge point cloud corresponding to the object, the vector calculation algorithm used in the prior art to calculate the tangent vector in three-dimensional space can be used to calculate the tangent vector of each 3D point in the edge point cloud, so as to generate point pair feature vectors based on the tangent vector. Those skilled in the art can select a vector calculation algorithm according to actual needs, and no specific limitation is made here.

[0030] In step S103 , for any two 3D points in the edge point cloud, a point pair including the two 3D points is constructed, and a point pair feature vector of the point pair is generated based on the tangent vectors of the two 3D points.

[0031] The edge point cloud corresponding to the object includes the posture information of each 3D point. For any two 3D points in the edge point cloud, a point pair containing the two 3D points is constructed, and a point pair feature vector of the point pair is generated based on the posture information of the two 3D points and the tangent vectors of the two 3D points, so that the object corresponds to multiple point pairs. Specifically, the point pair feature vector may include the Euclidean distance of the line vector between the two 3D points, the angle between the tangent vector of each 3D point in the two 3D points and the line vector, and the angle between the tangent vectors of the two 3D points. Those skilled in the art may also set the point pair feature vector to include other content according to actual needs, which is not limited here.

[0032] Step S104 , matching the edge point cloud corresponding to the object with the preset template point cloud according to the point pair feature vectors of each point pair of the object, and obtaining the posture information of the object so that the robot can perform a grasping operation according to the posture information of the object.

[0033] In order to conveniently and accurately identify the pose information of each object in the scene image, a template library containing multiple preset template point clouds has been pre-built. The preset template point clouds are point clouds corresponding to predetermined known objects that serve as matching benchmarks. For each object in the current scene, the edge point cloud corresponding to the object is matched with the preset template point cloud based on the point pair feature vectors of each point pair of the object, thereby obtaining the pose information of the object. The pose information of the object may specifically include the coordinate values ​​of the XYZ three-axis of the object's center in space and the XYZ three-axis direction of the object itself. After obtaining the pose information of the object, the pose information of the object can be transmitted to the robot so that the robot can perform a grasping operation on the object based on the pose information of the object.

[0034] According to the grasping method for objects with pure planar structures provided by this embodiment, the pure planar structure features are fully studied, the edge point cloud is extracted from the point cloud corresponding to the object, and the point pair feature vectors of the point pairs are generated based on the tangent vectors of the 3D points in the edge point cloud. The shape and structural features of the object are accurately reflected through the point pair feature vectors; the edge point cloud is matched with the preset template point cloud according to the point pair feature vector, and the posture information of the object is accurately identified, which helps the robot to accurately and firmly perform grasping operations according to the posture information of the object, avoiding grasping errors such as the robot being unable to successfully grasp the object or the object falling after being grasped; and, in this solution, only the edge point cloud corresponding to the object is extracted to participate in the matching process, and the non-edge point cloud does not participate in the matching process, thereby effectively reducing the matching workload and improving the recognition efficiency of the posture information of the object.

[0035] Figure 2 FIG. 4 shows a flow chart of a method for grasping an object with a purely planar structure according to another embodiment of the present invention. Figure 2 As shown, the method includes the following steps:

[0036] Step S201: Obtain a scene image of the current scene and a point cloud corresponding to the scene image, input the scene image into a trained deep learning segmentation model for instance segmentation processing, and obtain segmentation results of each object in the scene image.

[0037] The current scene contains a plurality of objects with purely planar structures. The scene image and depth image of the current scene can be collected by a camera arranged at an upper position, wherein the camera can be specifically a 3D camera, and the 3D camera can be arranged at an upper position, such as directly above or obliquely above, for simultaneously collecting information of the current scene within the camera's field of view to obtain a scene image and a depth image. Specifically, the 3D camera can include components such as laser detectors, LEDs and other visible light detectors, infrared detectors and / or radar detectors, and these components are used to detect the current scene to obtain a depth image. The scene image can specifically be an RGB image, and the pixels of the scene image and the depth image correspond one to one. By processing the scene image and the depth image, the point cloud corresponding to the scene image can be conveniently obtained. In step S201, the scene image of the current scene collected by the camera and the point cloud corresponding to the scene image obtained by processing the scene image and the depth image can be obtained.

[0038] In order to be able to conveniently and accurately segment each object contained in the scene image, sample scene images can be collected in advance, a training sample set can be constructed, and each sample scene image in the training sample set can be trained using a deep learning algorithm. Finally, a deep learning segmentation model is trained. Then, after obtaining the scene image of the current scene, the scene image can be input into the trained deep learning segmentation model, and a series of model calculations can be performed using the trained deep learning segmentation model to perform instance segmentation processing on each object contained in the scene image, thereby obtaining the segmentation results of each object in the scene image.

[0039] Step S202 : determining the point cloud corresponding to each object based on the point cloud corresponding to the scene image and the segmentation results of each object.

[0040] Among them, the segmentation results of each object may include a binary segmentation image of each object. For each object, the binary segmentation image of the object may include the object area where the object is located and the non-object area other than the object area. The object area can be specifically represented by a white area, and the non-object area can be specifically represented by a black area.

[0041] Then, for each object, the point cloud corresponding to the scene image can be projected into the binary segmentation image of the object, and the 3D points in the point cloud corresponding to the scene image projected into the object area of ​​the binary segmentation image are used as the 3D points corresponding to the object to obtain the point cloud corresponding to the object. Specifically, all 3D points in the point cloud corresponding to the scene image are projected. If a 3D point in the point cloud corresponding to the scene image falls into the white object area after projection, then the 3D point is considered to belong to the object, that is, the 3D point is the 3D point corresponding to the object. All 3D points corresponding to the object are summarized to obtain the point cloud corresponding to the object. Through this processing method, the point cloud corresponding to the object is accurately determined.

[0042] Step S203: Obtain point clouds corresponding to multiple objects in the current scene.

[0043] In step S204 , for each object, edge extraction is performed on the point cloud corresponding to the object to obtain the edge point cloud corresponding to the object, and the tangent vector of each 3D point in the edge point cloud is calculated.

[0044] In an optional embodiment, an edge point cloud can be extracted based on a 3D method. Specifically, for each 3D point in the point cloud corresponding to each object, the adjacent 3D points located in the preset neighborhood area of ​​the 3D point are searched in the point cloud to obtain a point set containing the 3D point and the adjacent 3D points. For example, a neighborhood radius (such as 1 cm or 0.3 cm, etc.) can be set, and the area with the 3D point as the center and the neighborhood radius as the radius is configured as the preset neighborhood area of ​​the 3D point. Then, the 3D points located in the preset neighborhood area of ​​the 3D point are searched in the point cloud corresponding to the object. The found 3D points are called adjacent 3D points, and the 3D point and all the adjacent 3D points of the 3D point are summarized to obtain a point set. After obtaining the point set, the normal vector calculation method in the prior art can be used to calculate the normal vector according to the points in the point set, and a plane where the normal vector is located can be set for the normal vector.

[0045] Then, a line is constructed between the 3D point and each adjacent 3D point in the point set, and the first angle between the projection line of each line in the plane where the normal vector is located and the specified reference direction axis is calculated. For example, each line is projected onto the plane where the normal vector is located to obtain the projection line corresponding to each line. The lines and projection lines correspond one to one, and the angle between each projection line and the specified reference direction axis (such as the X-axis, etc.) in the plane where the normal vector is located is calculated. In this embodiment, in order to facilitate the distinction between this angle and the angle between the tangent vector of each 3D point and the connecting line vector of any two 3D points in the edge point cloud below, and the angle between the tangent vectors of any two 3D points in the edge point cloud, the angle between the projection line and the specified reference direction axis in the plane where the normal vector is located is called the first angle, the angle between the tangent vector of the 3D point and the connecting line vector is called the second angle, and the angle between the tangent vectors of the two 3D points is called the third angle.

[0046] After obtaining the first angle corresponding to each projection line, each projection line can be sorted according to the first angle, for example, in a clockwise direction, a counterclockwise direction, an order from large to small first angle, or an order from small to large first angle, and then the angle difference between the first angles corresponding to the two adjacent projection lines after sorting is calculated, and the maximum value of the angle difference is selected to determine whether the maximum value of the angle difference is greater than a preset angle difference threshold; if so, it means that there is at least one area around the 3D point where no other 3D points exist, and the 3D point is determined as an edge point; if not, it means that there are other 3D points all around the 3D point, and the 3D point is determined as a non-edge point. The above processing is performed on each 3D point in the point cloud corresponding to the object, and whether it is an edge point is determined by comparing the maximum value of the angle difference with the preset angle difference threshold, and then all edge points of the object are summarized to obtain the edge point cloud corresponding to the object. Those skilled in the art can set the preset angle difference threshold according to actual needs, and no specific limitation is made here.

[0047] In another optional embodiment, the edge point cloud can also be extracted based on a 2D projection method. Specifically, each 3D point in the point cloud corresponding to the object can be projected onto a preset plane (such as the plane where the camera is located) to obtain a projection image. For example, in the projection image, the area where the 3D point projection exists is marked with white, and the area where the 3D point projection does not exist is marked with black. Then, the contour pixels in the projection image can be identified, and the 3D points corresponding to the contour pixels are determined as edge points. All edge points of the object are summarized to obtain the edge point cloud corresponding to the object. Among them, an image recognition algorithm can be used to identify the contour boundary line formed in the projection image, and the pixel points corresponding to the contour boundary line are determined as contour pixels. The 3D points projected onto the contour pixels in the point cloud corresponding to the object are determined as the 3D points corresponding to the contour pixels. That is to say, if a 3D point in the point cloud corresponding to the object falls on the position of the contour pixel after projection, then it is considered that the 3D point has a corresponding relationship with the contour pixel, and the 3D point is the 3D point corresponding to the contour pixel, and the 3D point is the edge point.

[0048] Both the 3D and 2D projection methods described above allow for the precise extraction of edge point clouds corresponding to objects. Once the edge point cloud corresponding to the object is extracted, the tangent vectors of each 3D point in the edge point cloud can be calculated using existing vector calculation algorithms for calculating tangent vectors in three-dimensional space, thereby generating point-pair feature vectors based on the tangent vectors.

[0049] In step S205 , for any two 3D points in the edge point cloud, a point pair including the two 3D points is constructed, and a point pair feature vector of the point pair is generated based on the tangent vectors of the two 3D points.

[0050] The edge point cloud corresponding to the object includes the pose information of each 3D point, including the coordinate values ​​of the 3D points in the X, Y, and Z axes of space. In step S205, a line vector between the two 3D points is constructed based on the coordinate values ​​of the 3D points in the X, Y, and Z axes of space, and the Euclidean distance of the line vector is calculated. Next, the second angle between the tangent vector of each 3D point and the line vector is calculated, as well as the third angle between the tangent vectors of the two 3D points. Finally, the Euclidean distance of the line vector, the second angle, and the third angle are used to generate a point pair feature vector for the point pair.

[0051] Assume that any two 3D points in the edge point cloud are represented as m1 and m2 respectively, the tangent vector of m1 is represented as t1, the tangent vector of m2 is represented as t2, and the line vector between m1 and m2 is represented as d, where d = m2-m1, then the point pair feature vector containing the point pair m1 and m2 can be expressed as (||d||2,∠(t1,d),∠(t2,d),∠(t1,t2)), where ||d||2 represents the Euclidean distance of the line vector d, ∠(t1,d) represents the second angle between the tangent vector t1 of m1 and the line vector d, ∠(t2,d) represents the second angle between the tangent vector t2 of m2 and the line vector d, and ∠(t1,t2) represents the third angle between the tangent vector t1 of m1 and the tangent vector t2 of m2.

[0052] In step S206 , the point pair feature vectors of each point pair in the edge point cloud corresponding to the object are matched for the first time with the point pair feature vectors of each point pair in the edge point cloud of the preset template point cloud to obtain a first matching result.

[0053] Among them, the preset template point cloud is a point cloud corresponding to a known object that is predetermined and serves as a matching reference. The edge point cloud of the preset template point cloud is an edge point cloud corresponding to the known object. The edge point cloud includes the posture information of each 3D point. For any two 3D points in the edge point cloud of the preset template point cloud, a point pair containing two 3D points is constructed, and a point pair feature vector of the point pair is generated based on the posture information of the two 3D points and the tangent vectors of the two 3D points, so that the preset template point cloud also corresponds to multiple point pairs.

[0054] After obtaining the point-pair feature vectors for each point pair in the edge point cloud corresponding to the object through processing in step S205, a first matching is performed on the point-pair feature vectors for each point pair in the edge point cloud corresponding to the object with the point-pair feature vectors for each point pair in the edge point cloud of the preset template point cloud to obtain a first matching result. The pose information of each 3D point is predefined in each preset template point cloud. During the first matching process, each preset template point cloud needs to be transformed into the current scene so that the edge point cloud of the preset template point cloud after the pose transformation overlaps as much as possible with the edge point cloud corresponding to the object in the current scene, thereby obtaining a first matching result, wherein the first matching result may include multiple pose transformation relationships of the matched preset template point cloud.

[0055] In step S207, the pose information of each 3D point in the edge point cloud corresponding to the object is matched for the second time with multiple pose transformation relationships of the matching preset template point cloud, and the pose information of the object is determined based on the pose transformation relationship of the matching preset template point cloud with the highest matching score in the second matching result.

[0056] Because the first matching result may include multiple pose transformation relationships of the matching preset template point cloud, in order to more accurately determine the object's pose information, in this embodiment, a second matching is performed between the pose information of each 3D point in the edge point cloud corresponding to the object and the multiple pose transformation relationships of the matching preset template point cloud. Specifically, a preset evaluation algorithm can be used to calculate the matching score between the pose information of each 3D point in the edge point cloud corresponding to the object and the multiple pose transformation relationships of the matching preset template point cloud, thereby obtaining the second matching result. Those skilled in the art can select the preset evaluation algorithm based on actual needs, and this is not limited here. For example, the preset evaluation algorithm can be an ICP (Iterative Closest Point) algorithm, a GMM (Gaussian Mixed Model) algorithm, etc. The second matching further optimizes and corrects the first matching result. The pose transformation relationship of the matching preset template point cloud with the highest matching score in the second matching result is used as the final matching object, and the pose information of the object is determined based on the pose transformation relationship of the matching preset template point cloud with the highest matching score in the second matching result.

[0057] Each object in the current scene is processed according to steps S204 to S207 to obtain the pose information of each object. Since the pose information of the object is determined in the camera coordinate system, in order to facilitate the robot to locate the object, it is necessary to use a preset conversion algorithm to convert the object's pose information to the robot coordinate system. The converted pose information of the object is then transmitted to the robot, and the robot performs a grasping operation on the object based on the converted pose information.

[0058] According to the grasping method for objects with purely planar structures provided by this embodiment, a deep learning segmentation model is used to perform instance segmentation on the scene image, thereby achieving accurate segmentation of each object in the scene image; edge point clouds are extracted from the point clouds corresponding to the objects based on 3D or 2D projection, thereby achieving accurate extraction of edge point clouds, and point pair feature vectors of point pairs are generated based on the tangent vectors of the 3D points in the edge point clouds, thereby achieving accurate reflection of the shape and structural features of the objects through the point pair feature vectors; based on the point pair feature vectors, the point clouds corresponding to each object are matched with the preset template point clouds for a second time, thereby achieving accurate recognition of the object's posture information, which helps the robot to accurately and firmly perform grasping operations based on the object's posture information; and, in this solution, only the edge point clouds corresponding to the objects are extracted to participate in the matching process, and non-edge point clouds do not participate in the matching process, thereby effectively reducing the matching workload and improving the recognition efficiency of the object's posture information.

[0059] Figure 3FIG. 1 shows a structural block diagram of a grasping device for a purely planar object according to an embodiment of the present invention. Figure 3 As shown, the device includes: a first acquisition module 301 , an edge extraction module 302 , a point pair construction module 303 and a matching module 304 .

[0060] The first acquisition module 301 is adapted to: acquire point clouds corresponding to a plurality of objects in a current scene; wherein the plurality of objects have a purely planar structure.

[0061] The edge extraction module 302 is adapted to: for each object, perform edge extraction on the point cloud corresponding to the object, obtain the edge point cloud corresponding to the object, and calculate the tangent vector of each 3D point in the edge point cloud.

[0062] The point pair construction module 303 is adapted to construct a point pair including the two 3D points for any two 3D points in the edge point cloud, and generate a point pair feature vector of the point pair based on the tangent vectors of the two 3D points.

[0063] The matching module 304 is adapted to match the edge point cloud corresponding to the object with the preset template point cloud according to the point pair feature vectors of each point pair of the object to obtain the posture information of the object so that the robot can perform a grasping operation according to the posture information of the object.

[0064] Optionally, the apparatus may further include a second acquisition module 305, an instance segmentation module 306, and an object point cloud determination module 307. The second acquisition module 305 is configured to acquire a scene image of the current scene and a point cloud corresponding to the scene image. The instance segmentation module 306 is configured to input the scene image into a trained deep learning segmentation model to perform instance segmentation processing and obtain segmentation results for each object in the scene image. The object point cloud determination module 307 is configured to determine the point cloud corresponding to each object based on the point cloud corresponding to the scene image and the segmentation results for each object.

[0065] Optionally, the edge extraction module 302 is further adapted to: for each 3D point in the point cloud, search for adjacent 3D points in the point cloud that are located within a preset neighborhood area of ​​the 3D point, obtain a point set containing the 3D point and the adjacent 3D points, and calculate a normal vector based on the point set; construct a connecting line between the 3D point and each adjacent 3D point in the point set, and calculate a first angle between the projection line of each connecting line in the plane where the normal vector is located and a specified reference direction axis; sort each projection line according to the first angle, and calculate the angle difference between the first angles corresponding to the two adjacent projection lines after sorting; determine whether the maximum value of the angle difference is greater than a preset angle difference threshold; if so, determine the 3D point as an edge point; if not, determine the 3D point as a non-edge point; summarize all edge points to obtain an edge point cloud corresponding to the object.

[0066] Optionally, the edge extraction module 302 is further adapted to: project each connecting line onto the plane where the normal vector is located to obtain a projection line corresponding to each connecting line; and calculate a first angle between each projection line and a specified reference direction axis in the plane where the normal vector is located.

[0067] Optionally, the edge extraction module 302 is further adapted to: project each 3D point in the point cloud corresponding to the object onto a preset plane to obtain a projected image; identify contour pixel points in the projected image, and determine the 3D points corresponding to the contour pixel points as edge points; and aggregate all edge points to obtain an edge point cloud corresponding to the object.

[0068] Optionally, the edge extraction module 302 is further adapted to: identify the contour boundary line formed in the projected image, determine the pixel points corresponding to the contour boundary line as contour pixel points, and determine the 3D points projected to the contour pixel points in the point cloud corresponding to the object as 3D points corresponding to the contour pixel points.

[0069] Optionally, the point pair construction module 303 is further adapted to: construct a line vector between two 3D points and calculate the Euclidean distance of the line vector; calculate the second angle between the tangent vector of each 3D point in the two 3D points and the line vector and the third angle between the tangent vectors of the two 3D points; and generate a point pair feature vector of the point pair using the Euclidean distance, the second angle and the third angle of the line vector.

[0070] Optionally, the matching module 304 is further adapted to: perform a first matching of the point pair feature vectors of each point pair in the edge point cloud corresponding to the object with the point pair feature vectors of each point pair in the edge point cloud of the preset template point cloud to obtain a first matching result; wherein the first matching result includes multiple pose transformation relationships of the matched preset template point cloud; perform a second matching of the pose information of each 3D point in the edge point cloud corresponding to the object with multiple pose transformation relationships of the matched preset template point cloud, and determine the pose information of the object based on the pose transformation relationship of the matched preset template point cloud with the highest matching score in the second matching result.

[0071] Optionally, the matching module 304 is further adapted to: calculate, by using a preset evaluation algorithm, a matching score between the pose information of each 3D point in the edge point cloud corresponding to the object and multiple pose transformation relationships of the matching preset template point cloud.

[0072] According to the grasping device for objects with purely planar structures provided by this embodiment, a deep learning segmentation model is used to perform instance segmentation on the scene image, thereby achieving accurate segmentation of each object in the scene image; edge point clouds are extracted from the point clouds corresponding to the objects based on 3D or 2D projection, thereby achieving accurate extraction of edge point clouds, and point pair feature vectors are generated based on the tangent vectors of the 3D points in the edge point clouds, thereby achieving accurate reflection of the shape and structural features of the objects through the point pair feature vectors; based on the point pair feature vectors, the point clouds corresponding to each object are matched with the preset template point clouds for a second time, thereby achieving accurate recognition of the object's posture information, which helps the robot to accurately and firmly perform grasping operations based on the object's posture information; and, in this solution, only the edge point clouds corresponding to the objects are extracted to participate in the matching process, and non-edge point clouds do not participate in the matching process, thereby effectively reducing the matching workload and improving the recognition efficiency of the object's posture information.

[0073] The present invention also provides a non-volatile computer storage medium storing at least one executable instruction, which can execute the method for grasping an object with a purely planar structure in any of the above method embodiments.

[0074] Figure 4 A schematic structural diagram of a computing device according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

[0075] like Figure 4 As shown, the computing device may include: a processor 402 , a communications interface 404 , a memory 406 , and a communication bus 408 .

[0076] in:

[0077] The processor 402 , the communication interface 404 , and the memory 406 communicate with each other via a communication bus 408 .

[0078] The communication interface 404 is used to communicate with other devices such as clients or other servers.

[0079] The processor 402 is configured to execute the program 410 , and specifically to execute the relevant steps in the above embodiment of the method for grasping an object with a purely planar structure.

[0080] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0081] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in a computing device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0082] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0083] Program 410 can specifically be used to cause processor 402 to execute the method for grasping a purely planar object in any of the above-described method embodiments. The specific implementation of each step in program 410 can be found in the corresponding descriptions of the corresponding steps and units in the above-described embodiments for grasping a purely planar object, and will not be repeated here. Those skilled in the art will clearly understand that, for ease and brevity of description, the specific operating processes of the devices and modules described above can refer to the corresponding process descriptions in the above-described method embodiments, and will not be repeated here.

[0084] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general-purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the best mode of the present invention.

[0085] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0086] Similarly, it should be understood that in order to streamline the present disclosure and aid understanding of one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

[0087] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0088] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0089] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in accordance with the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing a portion or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0090] It should be noted that the above embodiments illustrate rather than limit the invention, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A method for grasping an object with a purely planar structure, the method comprising: Obtaining point clouds corresponding to a plurality of objects in a current scene; wherein the plurality of objects have a purely planar structure; For each object, perform edge extraction on the point cloud corresponding to the object to obtain the edge point cloud corresponding to the object, and calculate the tangent vector of each 3D point in the edge point cloud; For any two 3D points in the edge point cloud, construct a point pair containing the two 3D points, and generate a point pair feature vector of the point pair based on the tangent vectors of the two 3D points; According to the point pair feature vectors of each point pair of the object, the edge point cloud corresponding to the object is matched with the preset template point cloud to obtain the pose information of the object so that the robot can perform the grasping operation according to the pose information of the object; The step of performing edge extraction on the point cloud corresponding to each object to obtain the edge point cloud corresponding to the object further includes: For each 3D point in the point cloud, searching for adjacent 3D points in a preset neighborhood of the 3D point in the point cloud to obtain a point set including the 3D point and the adjacent 3D points, and calculating a normal vector based on the point set; Constructing a line between the 3D point and each adjacent 3D point in the point set, and calculating a first angle between a projection line of each line in the plane where the normal vector lies and a specified reference direction axis; Sort the projection lines according to the first angle, and calculate the angle difference between the first angles corresponding to two adjacent projection lines after sorting; Determine whether the maximum value of the angle difference is greater than a preset angle difference threshold; if so, determine the 3D point as an edge point; if not, determine the 3D point as a non-edge point; Summarize all edge points to obtain the edge point cloud corresponding to the object.

2. The method according to claim 1, wherein Before obtaining point clouds corresponding to multiple objects in the current scene, the method further includes: Obtain a scene image of the current scene and a point cloud corresponding to the scene image, input the scene image into a trained deep learning segmentation model for instance segmentation processing, and obtain segmentation results for each object in the scene image; The point cloud corresponding to each object is determined based on the point cloud corresponding to the scene image and the segmentation results of each object.

3. The method according to claim 1, wherein The step of calculating the first angle between the projection line of each connecting line in the plane where the normal vector is located and the specified reference direction axis further comprises: Projecting each connecting line onto the plane where the normal vector is located to obtain a projection line corresponding to each connecting line; Calculate the first angle between each projection line and the specified reference direction axis in the plane where the normal vector is located.

4. The method according to claim 1, wherein The step of performing edge extraction on a point cloud corresponding to each object to obtain an edge point cloud corresponding to the object further includes: Project each 3D point in the point cloud corresponding to the object onto a preset plane to obtain a projected image; Identifying contour pixel points in the projected image, and determining 3D points corresponding to the contour pixel points as edge points; Summarize all edge points to obtain the edge point cloud corresponding to the object.

5. The method according to claim 4, wherein The identifying of contour pixels in the projection image further comprises: A contour boundary line formed in the projected image is identified, a pixel point corresponding to the contour boundary line is determined as a contour pixel point, and a 3D point projected to the contour pixel point in the point cloud corresponding to the object is determined as a 3D point corresponding to the contour pixel point.

6. The method according to any one of claims 1 to 5, wherein: Generating the point pair feature vector of the point pair based on the tangent vectors of the two 3D points further includes: Constructing a line vector between the two 3D points and calculating the Euclidean distance of the line vector; Calculating a second angle between a tangent vector of each of the two 3D points and the connecting vector and a third angle between the tangent vectors of the two 3D points; A point pair feature vector of the point pair is generated using the Euclidean distance of the line vector, the second angle, and the third angle.

7. The method according to any one of claims 1 to 5, wherein: The step of matching the edge point cloud corresponding to the object with a preset template point cloud based on the point pair feature vectors of each point pair of the object to obtain the pose information of the object further includes: Performing a first matching of the point pair feature vectors of each point pair in the edge point cloud corresponding to the object with the point pair feature vectors of each point pair in the edge point cloud of the preset template point cloud to obtain a first matching result; wherein the first matching result includes multiple pose transformation relationships of the matched preset template point cloud; The pose information of each 3D point in the edge point cloud corresponding to the object is matched for the second time with multiple pose transformation relationships of the matching preset template point cloud, and the pose information of the object is determined based on the pose transformation relationship of the matching preset template point cloud with the highest matching score in the second matching result.

8. The method according to claim 7, wherein: The second matching of the pose information of each 3D point in the edge point cloud corresponding to the object with multiple pose transformation relationships of the matching preset template point cloud further includes: Using a preset evaluation algorithm, the matching score between the pose information of each 3D point in the edge point cloud corresponding to the object and the multiple pose transformation relationships of the matching preset template point cloud is calculated.

9. A gripping device for an object with a purely planar structure, the device comprising: A first acquisition module is adapted to acquire point clouds corresponding to a plurality of objects in a current scene; wherein the plurality of objects have a purely planar structure; An edge extraction module is adapted to perform edge extraction on the point cloud corresponding to each object, obtain the edge point cloud corresponding to the object, and calculate the tangent vector of each 3D point in the edge point cloud; a point pair construction module adapted to construct a point pair containing any two 3D points in the edge point cloud, and generate a point pair feature vector of the point pair based on the tangent vectors of the two 3D points; A matching module is adapted to match the edge point cloud corresponding to the object with a preset template point cloud based on the point pair feature vectors of each point pair of the object, to obtain the pose information of the object so that the robot can perform a grasping operation based on the pose information of the object; Wherein, the edge extraction module is further adapted to: For each 3D point in the point cloud, searching for adjacent 3D points in a preset neighborhood of the 3D point in the point cloud to obtain a point set including the 3D point and the adjacent 3D points, and calculating a normal vector based on the point set; Constructing a line between the 3D point and each adjacent 3D point in the point set, and calculating a first angle between a projection line of each line in the plane where the normal vector lies and a specified reference direction axis; Sort the projection lines according to the first angle, and calculate the angle difference between the first angles corresponding to two adjacent projection lines after sorting; Determine whether the maximum value of the angle difference is greater than a preset angle difference threshold; if so, determine the 3D point as an edge point; if not, determine the 3D point as a non-edge point; Summarize all edge points to obtain the edge point cloud corresponding to the object.

10. The device according to claim 9, wherein The device further comprises: A second acquisition module, adapted to acquire a scene image of the current scene and a point cloud corresponding to the scene image; An instance segmentation module, adapted to input the scene image into a trained deep learning segmentation model to perform instance segmentation processing and obtain segmentation results of each object in the scene image; The object point cloud determination module is adapted to determine the point cloud corresponding to each object based on the point cloud corresponding to the scene image and the segmentation results of each object.

11. The device according to claim 9, wherein The edge extraction module is further adapted to: Projecting each connecting line onto the plane where the normal vector is located to obtain a projection line corresponding to each connecting line; Calculate the first angle between each projection line and the specified reference direction axis in the plane where the normal vector is located.

12. The device according to claim 9, wherein The edge extraction module is further adapted to: Project each 3D point in the point cloud corresponding to the object onto a preset plane to obtain a projected image; Identifying contour pixel points in the projected image, and determining 3D points corresponding to the contour pixel points as edge points; Summarize all edge points to obtain the edge point cloud corresponding to the object.

13. The device according to claim 12, wherein The edge extraction module is further adapted to: A contour boundary line formed in the projected image is identified, a pixel point corresponding to the contour boundary line is determined as a contour pixel point, and a 3D point projected to the contour pixel point in the point cloud corresponding to the object is determined as a 3D point corresponding to the contour pixel point.

14. The device according to any one of claims 9 to 13, wherein: The point pair building block is further adapted to: Constructing a line vector between the two 3D points and calculating the Euclidean distance of the line vector; Calculating a second angle between a tangent vector of each of the two 3D points and the connecting vector and a third angle between the tangent vectors of the two 3D points; A point pair feature vector of the point pair is generated using the Euclidean distance of the line vector, the second angle, and the third angle.

15. The device according to any one of claims 9 to 13, wherein: The matching module is further adapted to: Performing a first matching of the point pair feature vectors of each point pair in the edge point cloud corresponding to the object with the point pair feature vectors of each point pair in the edge point cloud of the preset template point cloud to obtain a first matching result; wherein the first matching result includes multiple pose transformation relationships of the matched preset template point cloud; The pose information of each 3D point in the edge point cloud corresponding to the object is matched for the second time with multiple pose transformation relationships of the matching preset template point cloud, and the pose information of the object is determined based on the pose transformation relationship of the matching preset template point cloud with the highest matching score in the second matching result.

16. The device according to claim 15, wherein The matching module is further adapted to: Using a preset evaluation algorithm, the matching score between the pose information of each 3D point in the edge point cloud corresponding to the object and the multiple pose transformation relationships of the matching preset template point cloud is calculated.

17. A computing device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute an operation corresponding to the method for grasping an object with a purely planar structure according to any one of claims 1 to 8.

18. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, wherein the executable instruction enables a processor to execute operations corresponding to the method for grasping an object with a purely planar structure according to any one of claims 1 to 8.

Images