Cartridge grasping method, device and electronic equipment

Abstract

The present disclosure provides a box grabbing method, device and electronic equipment. The box grabbing method comprises: obtaining initial three-dimensional point cloud information of a target box; processing the initial three-dimensional point cloud information to obtain an overall two-dimensional mask image of the target box on a preset surface; performing corner point detection on the overall two-dimensional mask image to obtain corner point parameters of the overall two-dimensional mask image; determining a two-dimensional mask image of at least one target box according to a preset shape and the obtained corner point parameters of the overall two-dimensional mask image, wherein the preset shape at least contains the shape of the grabbing surface of the target box; and controlling a grabbing device to grab the corresponding target box according to the two-dimensional mask image of the at least one target box. The present disclosure can efficiently and accurately grab each box when grabbing the box.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a box grabbing method and device and electronic equipment. BACKGROUND

[0002] In the application of box grabbing, the three-dimensional point cloud information of the box needs to be collected by the camera, and then the grabbing device is controlled to grab the box. However, in actual application, the three-dimensional point cloud information of at least two boxes may be collected by the camera at a time, which may cause the grabbing device to fail to grab the boxes one by one.

[0003] Based on the above technical problem, in the related art, a large amount of data is collected to train a neural network model, and then the trained neural network model is used to segment the three-dimensional point cloud information of the box to obtain the three-dimensional point cloud information of the box. However, this method requires a large amount of training data to be collected in advance, and the process is complicated. SUMMARY

[0004] The aspects of the present disclosure provide a box grabbing method, device and electronic equipment to solve the problem of complicated process and time-consuming in obtaining the three-dimensional point cloud information of the box.

[0005] The first aspect of the embodiment of the present disclosure provides a box grabbing method, comprising: obtaining initial three-dimensional point cloud information of a target box; processing the initial three-dimensional point cloud information to obtain an overall two-dimensional mask image of the target box on a preset surface; performing corner point detection on the overall two-dimensional mask image to obtain corner point parameters of the overall two-dimensional mask image; determining a two-dimensional mask image of at least one target box according to a preset shape and the obtained corner point parameters of the overall two-dimensional mask image, wherein the preset shape at least contains the shape of the grabbing surface of the target box; and controlling a grabbing device to grab the corresponding target box according to the two-dimensional mask image of the at least one target box.

[0006] The second aspect of the embodiment of the present disclosure provides a box grabbing device, comprising:

[0007] An obtaining module is configured to obtain initial three-dimensional point cloud information of a target box;

[0008] A processing module is configured to process the initial three-dimensional point cloud information to obtain an overall two-dimensional mask image of the target box on a preset surface;

[0009] A detection module is configured to perform corner point detection on the overall two-dimensional mask image to obtain corner point parameters of the overall two-dimensional mask image;

[0010] A determination module is configured to determine a two-dimensional mask image of at least one target box according to a preset shape and the obtained corner point parameters of the overall two-dimensional mask image;

[0011] The control module is configured to control the grabbing device to grab the corresponding target box body according to the two-dimensional mask image of the at least one target box body.

[0012] The third aspect of the embodiments of the present disclosure provides an electronic device, which comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the box body grabbing method of the first aspect when executing the computer program.

[0013] The fourth aspect of the present disclosure provides a computer-readable storage medium, and the computer-readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the box body grabbing method of the first aspect.

[0014] The fifth aspect of the present disclosure provides a computer program product, and the program product comprises a computer program stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to enable the electronic device to execute the box body grabbing method of the first aspect.

[0015] The embodiments of the present disclosure are applied to the scenarios of loading and unloading box bodies, and the initial three-dimensional point cloud information of a target box body is acquired; the initial three-dimensional point cloud information is processed to obtain an overall two-dimensional mask image of the target box body on a preset surface; an angle point detection module is configured to detect the overall two-dimensional mask image to obtain angle point parameters of the overall two-dimensional mask image; a determination module is configured to determine a two-dimensional mask image of at least one target box body according to a preset shape and the angle point parameters of the overall two-dimensional mask image; and a control module is configured to control a grabbing device to grab a corresponding target box body according to the two-dimensional mask image of the at least one target box body. When the box body is grabbed, each box body can be efficiently and accurately grabbed. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are included to provide a further understanding of the present disclosure and constitute a part of the present disclosure, illustrate the exemplary embodiments of the present disclosure and their descriptions serve to explain the present disclosure, but do not limit the present disclosure. In the drawings:

[0017] Figure 1 A schematic diagram of a box body grabbing method provided by the related art;

[0018] Figure 2 An application scenario diagram of a box body grabbing method provided by the exemplary embodiments of the present disclosure;

[0019] Figure 3 A step flowchart of a box body grabbing method provided by the exemplary embodiments of the present disclosure;

[0020] Figure 4A schematic diagram of a box grabbing method provided for an exemplary embodiment of the present disclosure;

[0021] Figure 5 A schematic diagram of another box grabbing method provided for an exemplary embodiment of the present disclosure;

[0022] Figure 6 A step flow chart of another box grabbing method provided for an exemplary embodiment of the present disclosure;

[0023] Figure 7 A schematic diagram of an outlier removal provided for an exemplary embodiment of the present disclosure;

[0024] Figure 8 A structural block diagram of a box grabbing device provided for an exemplary embodiment of the present disclosure;

[0025] Figure 9 A structural schematic diagram of an electronic device provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to make the purposes, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be described clearly and completely below in combination with specific embodiments of the present disclosure and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present disclosure.

[0027] In the related art, the three-dimensional point cloud information of the box captured by the industrial three-dimensional camera is mapped into a two-dimensional color image, and then the edge of the box is detected through the two-dimensional color image to perform the grabbing of the box. Exemplarily, referring to Figure 1 , the two-dimensional color image 12 corresponding to the box 11, wherein the patterns and lines in the two-dimensional color image will also be detected, which will interfere with the edge detection of the box, so that the obtained edge of the box is inaccurate, such as 13, and further affect the grabbing of the box.

[0028] To address the aforementioned issues, this disclosure addresses the following: First, it acquires initial 3D point cloud information of the target box; processes the initial 3D point cloud information to obtain a 2D mask image of the target box on a preset surface; a detection module performs corner detection on the 2D mask image to obtain corner parameters; a determination module determines at least one 2D mask image of the target box based on a preset shape and the acquired corner parameters; and a grasping device is controlled to grasp the corresponding target box based on the 2D mask images of at least one target box. Firstly, this disclosure can acquire the 3D point cloud information of the target box in real time to obtain a 2D mask image, enabling the grasping of the target box without pre-training a grasping model, making the process simple and efficient. Secondly, this disclosure maps the initial 3D point cloud information to obtain at least one 2D mask image of the target box, without requiring pattern or line information of the target box, thus avoiding interference from patterns and lines on the shape of the target box. Finally, by employing corner detection, it can accurately obtain the 2D mask image of at least one target box, improving the accuracy of subsequent target box grasping.

[0029] In this embodiment, the overall box-grabbing method can be implemented using a cloud computing system or electronic device. Furthermore, the server executing the box-grabbing method can be a cloud server, leveraging the advantages of cloud resources to run various algorithms; however, the box-grabbing method can also be applied to conventional servers or server arrays, and is not limited thereto.

[0030] Furthermore, one application scenario of this disclosure embodiment is as follows: Figure 2 , Figure 2 The system includes a gripping device 21 and a conveying device 12. Multiple boxes T1 to T4 are mounted on the conveying device 12. An industrial 3D camera (not shown) is mounted on the gripping device 11. The workpiece 3D camera acquires 3D point cloud information from the boxes entering its field of view. The boxes on the conveying device 12 can be a single box entering the field of view of the industrial 3D camera, or multiple boxes can enter the field of view of the industrial 3D camera simultaneously. For example, referring to… Figure 2 The acquisition view of the 3D camera on the workpiece is as shown in region A. Box T1 and box T4 will enter the acquisition view A as one, while box T2 and box T3 will enter the acquisition view A together.

[0031] During the box grasping process, the target box's pose and size information are located by acquiring its 3D point cloud information, enabling accurate grasping. In this disclosure, the box can be any shape of packaging box, such as a cube, cuboid, or irregularly shaped box, and one side of the box can be a polygon.

[0032] also, Figure 2Just an example of an application scenario, the embodiment of the disclosure can be applied to any grabbing scenario of the box body. The embodiment of the disclosure does not limit the specific application scenario.

[0033] Figure 3 A step flow chart of a box body grabbing method provided for an example embodiment of the disclosure. As shown in the box body grabbing method, the following steps are specifically included: Figure 3

[0034] S301, obtaining initial three-dimensional point cloud information of a target box body.

[0035] Wherein, the target box body can be one or more, referring to Figure 2 , the target box body can be a single box body T1 or a single box body T4, or two such as box body T2 and box body T3.

[0036] Further, the initial three-dimensional point cloud information of the target box body can be that the industrial three-dimensional camera shoots the target box body entering the shooting angle, and the initial three-dimensional point cloud information obtained by shooting includes the three-dimensional point cloud information of the preset surface, and can also include the three-dimensional point cloud information of other surfaces connected with the preset surface.

[0037] In the actual shooting process of the industrial three-dimensional camera, the point cloud information of the surface far away from the conveying device in the target box body is collected. Referring to Figure 4 , if the target box body T1 entering the shooting angle is a target box body T1, the industrial three-dimensional camera collects the point cloud information of the upper surface m1 of the target box body T1, wherein the side surface m2 and the side surface m3 of the target box body T1 can also be collected, and the initial three-dimensional point cloud information 41 is obtained, in the initial three-dimensional point cloud information 41, the point cloud information D1 corresponds to the upper surface m1, the point cloud information D2 corresponds to the side surface m2, and the point cloud information D3 corresponds to the side surface m3. The initial three-dimensional point cloud information 41 includes a plurality of points, each point has three-dimensional coordinates (x, y, z) and normal information (N). Wherein, 42 is the left view of the initial three-dimensional point cloud information 41, N1 is the normal information of the point cloud information D1, N2 is the normal information of the point cloud information D2, and N3 is the normal information of the point cloud information D3.

[0038] Exemplarily, if multiple target box bodies enter the shooting angle, referring to Figure 5 ​When the target box T2, the target box T3 and the target box T4 are in the field of view of the industrial three-dimensional camera, the industrial three-dimensional camera collects the point cloud information of the upper surface m4 of the target box T2, the target box T3 and the target box T4. In addition, the side surface m5 and the side surface m7 of the target box T2 and the side surface m6 and the side surface m8 of the target box T3 may also be collected. The initial three-dimensional point cloud information 51 is obtained, in which the point cloud information D4 corresponds to the upper surface m4, the point cloud information D5 corresponds to the side surface m5, the point cloud information D6 corresponds to the side surface m6, the point cloud information D7 corresponds to the side surface m7, and the point cloud information D8 corresponds to the side surface m8. The initial three-dimensional point cloud information 51 includes a plurality of points, each of which has three-dimensional coordinates (x, y, z) and normal information (N). The left view of the initial three-dimensional point cloud information 51 is 52, the normal information of the point cloud information D4 is N4, the normal information of the point cloud information D5 is N5, the normal information of the point cloud information D6 is N6, the normal information of the point cloud information D7 is N7, and the normal information of the point cloud information D8 is N8.

[0039] In the present disclosure, the initial three-dimensional point cloud information can include the three-dimensional point cloud information of the plurality of surfaces of the target box.

[0040] S302, processing the initial three-dimensional point cloud information to obtain the overall two-dimensional mask image of the target box on the preset surface.

[0041] The preset surface is a surface away from the conveying device, for example Figure 4 the upper surface m1 in Figure 5 the upper surface m4 in

[0042] In the present disclosure, the three-dimensional point cloud information of the preset surface is extracted from the initial three-dimensional point cloud information, such as Figure 4 the point cloud information D1 in Figure 5 the point cloud information D5 in

[0043] In addition, each pixel point in the overall two-dimensional mask image is white, that is, the RGB value of each pixel point is (255, 255, 255). Each pixel point has a coordinate value (x, y).

[0044] S303, corner point detection is performed on the overall two-dimensional mask image to obtain the corner point parameters of the overall two-dimensional mask image.

[0045] The corner point detection can be performed by determining the point with the maximum curvature on the image edge curve in the overall two-dimensional mask image as the corner point. In the present disclosure, the corner point of the overall two-dimensional mask image can also be determined by other methods, which are not limited herein.

[0046] Furthermore, the obtained corner point parameters include: the number of corner points and the position information of each corner point.

[0047] For example, refer to Figure 4 Corner detection was performed on the overall 2D mask image Y1 to obtain four corner points. (Refer to...) Figure 5 Corner detection was performed on the overall two-dimensional mask image Y3 to obtain 12 corner points (t1 to t12).

[0048] S304, Based on the preset shape and the corner point parameters of the acquired overall two-dimensional mask image, determine at least one two-dimensional mask image of the target box.

[0049] The preset shape includes at least the shape of the gripping surface of the target box.

[0050] In this disclosure, the preset shape is the shape of one surface of the target box. The preset shape can be stored in the storage space in advance. After detecting the corner points of the overall two-dimensional mask image, multiple corner points can be connected in a combination manner. If the area formed by connecting multiple corner points is the same as the preset shape, then the area enclosed by the multiple corner points is used as the two-dimensional mask image of at least one target box.

[0051] Furthermore, after detecting the corner points, the area enclosed by the corner points that is the same as or similar to a preset shape is determined as a two-dimensional mask image of at least one target box.

[0052] For example, refer to Figure 4 Corner detection is performed on the overall 2D mask image Y1 to obtain four corner points. The shape enclosed by these four corner points is the same as the preset shape. Therefore, in the overall 2D mask image Y1, the area enclosed by these four corner points constitutes the 2D mask image Y2 of at least one target box. (Refer to...) Figure 5 Corner detection was performed on the overall 2D mask image Y3 to obtain 12 corner points (t1 to t12). Among them, the region enclosed by t1, t2, t4, and t5 is the 2D mask image of at least one target box (Y4). The region enclosed by t3, t6, t7, and t8 is the 2D mask image of another at least one target box (Y5).

[0053] In another alternative embodiment, refer to Figure 5 After obtaining two-dimensional mask images Y4 and Y5, the shape of the two-dimensional mask image Y6 formed by the remaining corner points (t9 to t12) is different from the preset shape. It can be determined that the box T4 corresponding to the two-dimensional mask image Y6 does not conform to the grasping pose. Therefore, the box T4 can not be grasped and can be recycled by the conveyor belt for re-sorting, or an alarm can be set to prompt the staff to process the box T4.

[0054] S305, controlling the grabbing device to grab the corresponding target box body according to the two-dimensional mask image of the at least one target box body.

[0055] In the present disclosure, after obtaining the two-dimensional mask image of the at least one target box body, the pose information and the size information of the target box body can be determined, and then accurate grabbing of the target box body is realized.

[0056] For example, referring to Figure 4 and Figure 2 , after determining the pose information and the size information of the box body T1, the box body T1 can be grabbed. Referring to Figure 5 and Figure 2 , the two-dimensional mask image Y4 of the at least one target box body can be used to control the grabbing device to accurately grab the box body T2, and the two-dimensional mask image Y5 of the at least one target box body can be used to control the grabbing device to accurately grab the box body T3.

[0057] In the present disclosure, when only one target box body enters the grabbing range, the target box body can be accurately grabbed, and when multiple target box bodies are placed closely and enter the grabbing range, the grabbing device can be controlled to accurately grab the target box bodies one by one, so that the present disclosure can realize single accurate grabbing of the target box body at a time, avoiding grabbing errors. In addition, since the present disclosure uses a two-dimensional mask image for grabbing the target box body, the influence of the pattern on the target box body on grabbing can be avoided.

[0058] Further, on the basis of the above-mentioned embodiments, the present disclosure provides another box body grabbing method, referring to Figure 6 , which specifically includes the following steps:

[0059] S601, controlling an industrial three-dimensional camera to collect point cloud information of a target box body to obtain initial three-dimensional point cloud information.

[0060] The target box body is placed on a conveying device, and the target box body is conveyed by the conveying device into the collection angle of the industrial three-dimensional camera. For example, referring to Figure 2 , the conveying device 22 can be a conveyor belt. The collection angle is, for example, the area A.

[0061] In the present disclosure, the industrial three-dimensional camera uses a camera model, a binocular system and a point cloud model to realize point cloud collection of the target box body, and the specific implementation process is not limited herein. The obtained initial three-dimensional point cloud information includes a plurality of points, each point having three-dimensional coordinates and normal information.

[0062] In addition, the industrial three-dimensional camera collects point cloud information of a preset surface of the target box body, and when the preset surface is collected, point cloud information of other surfaces can be collected.

[0063] S602, preprocessing the initial three-dimensional point cloud information to obtain target three-dimensional point cloud information.

[0064] The target three-dimensional point cloud information includes three-dimensional point cloud information of the target box body on the preset surface. The purpose of preprocessing the initial three-dimensional point cloud information is to remove three-dimensional point cloud information of other surfaces outside the preset surface and remove outliers of the preset surface. The initial three-dimensional point cloud information can be preprocessed by point cloud clustering and outlier removal to obtain the target three-dimensional point cloud information.

[0065] Specifically, preprocessing the initial three-dimensional point cloud information to obtain target three-dimensional point cloud information includes: using clustering analysis to delete three-dimensional point cloud information not belonging to the preset surface in the initial three-dimensional point cloud information to obtain target three-dimensional point cloud information of the preset surface; and / or removing outliers in the initial three-dimensional point cloud information to obtain the target three-dimensional point cloud information.

[0066] In an optional embodiment, the clustering analysis method can be used to delete three-dimensional point cloud information not belonging to the preset surface in the initial three-dimensional point cloud information to obtain intermediate three-dimensional point cloud information of the preset surface, and then outliers in the intermediate three-dimensional point cloud information are removed to obtain the target three-dimensional point cloud information.

[0067] In another optional embodiment, outliers in the initial three-dimensional point cloud information can be removed to obtain intermediate three-dimensional point cloud information, and then the clustering analysis method is used to delete three-dimensional point cloud information not belonging to the preset surface in the intermediate three-dimensional point cloud information to obtain target three-dimensional point cloud information of the preset surface.

[0068] In the present disclosure, the above method can be used to obtain target three-dimensional point cloud information that is uniformly distributed and only has the preset surface.

[0069] The deleting of three-dimensional point cloud information not belonging to the preset surface to obtain target three-dimensional point cloud information of the preset surface includes: determining normal information of each point in the initial three-dimensional point cloud information; clustering points belonging to the same normal information in the initial three-dimensional point cloud information to obtain multiple clusters; determining a cluster including the most points as a target cluster; and determining three-dimensional point cloud information belonging to the target cluster as target three-dimensional point cloud information of the preset surface.

[0070] Reference Figure 4 In the initial three-dimensional point cloud information, points with the same normal information are clustered. The normal information of points in point cloud information D1 is N1, so they are clustered into one class. The normal information of points in point cloud information D2 is N2, so they are clustered into one class. The normal information of points in point cloud information D3 is N3, so they are clustered into one class. Among them, point cloud information D1 includes the most points, so the cluster corresponding to point cloud information D1 is the target cluster, and point cloud information D1 is the target three-dimensional point cloud information.

[0071] Further, Figure 5 The manner of determining the target three-dimensional point cloud information is the same as Figure 4 The points with normal information N4 are clustered into one group. The points with normal information N5 are clustered into one group. The points with normal information N6 are clustered into one group. The points with normal information N7 are clustered into one group. The points with normal information N8 are clustered into one group. Further, the point cloud information D4 is determined as the target three-dimensional point cloud information.

[0072] Further, the outlier points are removed, that is, the points far away from the point group in the initial three-dimensional point cloud information are removed. For example, referring to Figure 7 , the points in the region L are the outlier points. The outlier points are removed to obtain the target three-dimensional point cloud information with higher quality. Figure 5 , the points in the region L are the outlier points. The outlier points are removed to obtain the target three-dimensional point cloud information with higher quality.

[0073] In addition, the disclosure can also perform other processing on the points in the initial three-dimensional point cloud information, so that the points in the target three-dimensional point cloud information are more uniformly distributed. The specific processing manner is not limited herein.

[0074] S603, mapping the target three-dimensional point cloud information into an overall two-dimensional mask image.

[0075] The target three-dimensional point cloud information is mapped into an overall two-dimensional mask image, including: projecting the target three-dimensional point cloud information into two-dimensional information; determining the region corresponding to the two-dimensional information as the overall two-dimensional mask image, and each pixel point in the overall two-dimensional mask image is white and has corresponding two-dimensional information.

[0076] In the target three-dimensional point cloud information, each point has three-dimensional coordinates (x, y, z), and the two-dimensional information projected from the three-dimensional coordinates (x, y, z) is the two-dimensional coordinates (x, y). The projection can be along the normal of the plane where the target three-dimensional point cloud information is located, and the projection is performed to the plane perpendicular to the normal. The region where the projected two-dimensional information is located is determined as the two-dimensional mask image.

[0077] S604, performing corner point detection on the overall two-dimensional mask to obtain the corner point parameters of the overall two-dimensional mask image.

[0078] The specific corner point detection manner is described in the above embodiment content, which will not be repeated here.

[0079] S605, according to a preset shape, performing segmentation and connection between the corner points of the overall two-dimensional mask image.

[0080] Wherein, the shape of the connection of the plurality of target corner points in each corner point is similar to the preset shape; wherein, the preset shape is the shape of one surface of the target box body, the shape surrounded by the plurality of corner points is matched with the preset shape, if similar, the two-dimensional mask image surrounded by the plurality of corner points is determined as the two-dimensional mask image of the target box body.

[0081] Exemplarily, referring to Figure 4 , all the corner points in the overall two-dimensional mask image Y1 are target corner points. Referring to Figure 5 , the corner point t1, the corner point t2, the corner point t4 and the corner point t5 are connected in the overall two-dimensional mask image Y3. The corner point t3, the corner point t6, the corner point t7 and the corner point t8 are connected.

[0082] S606, screening the corner point surrounding shape consistent with the preset shape as the two-dimensional mask image of each target box body.

[0083] Further, if the corner point surrounding shape obtained by connecting the remaining corner points is inconsistent with the preset shape, re-segmentation or alarm processing is performed. For example, in Figure 5 , the corner point t3, the corner point t6, the corner point t7 and the corner point t8 are connected to obtain a two-dimensional mask image Y5 of the preset shape as the two-dimensional mask image of the target box body, then the corner point t1, the corner point t2, the corner point t4 and the corner point t5 are connected to obtain another two-dimensional mask image Y4 of the preset shape, and the corner point surrounding shape obtained by connecting the remaining t9, the corner point t10, the corner point t11 and the corner point t12 is inconsistent with the preset shape, then alarm processing can be performed.

[0084] Wherein, the end-to-end connection of each corner point in a group of target corner points obtains at least one two-dimensional mask image of the target box body. In the present disclosure, for the case of multiple target box bodies, the method is segmentation of the overall two-dimensional mask image to obtain the two-dimensional mask image of a single target box body, so that the fitted target box body can be separated and grasped respectively.

[0085] S607, according to the two-dimensional mask image of at least one target box body, the three-dimensional point cloud information of a single target box body is extracted from the target three-dimensional point cloud information.

[0086] Wherein, after obtaining the two-dimensional mask image of a single target box body, the three-dimensional point cloud information of a single target box body can be matched and extracted from the target three-dimensional point cloud information according to the two-dimensional mask image.

[0087] Exemplarily, referring to Figure 4 , the two-dimensional mask image Y1 of at least one target box body is matched to obtain the three-dimensional point cloud information D1 of a single target box body in the target three-dimensional point cloud information. Referring to Figure 5, the two-dimensional mask image Y4 of the at least one target box, and the three-dimensional point cloud information D5 of the single target box matched in the target three-dimensional point cloud information D4. The two-dimensional mask image Y5 of the shape, and the three-dimensional point cloud information D6 of the single target box matched in the target three-dimensional point cloud information D4.

[0088] S608, determining the pose information and the size information of the target box according to the three-dimensional point cloud information of the single target box.

[0089] The three-dimensional point cloud information of the single target box includes a plurality of points, each point having three-dimensional coordinates and normal information, and the pose information and the size information of the target box can be determined according to the three-dimensional coordinates.

[0090] S609, controlling the grabbing device to grab the corresponding single target box according to the pose information and the size information.

[0091] The grabbing device grabs the single target box according to the pose information and the size information.

[0092] In the present disclosure, when only one target box enters the grabbing range, the target box can be accurately grabbed. When there are multiple target boxes and the multiple target boxes are closely placed and enter the grabbing range, the multiple target boxes are difficult to distinguish on the point cloud, but the two-dimensional mask image corresponding to the single target box can be extracted through the corner point detection of the two-dimensional mask image, and the point cloud of the single target box is segmented out, so that accurate grabbing is realized.

[0093] In the embodiments of the present disclosure, referring to Figure 8 In addition to providing the box grabbing method, the present disclosure also provides a box grabbing device 80 for executing the above-mentioned box grabbing method. The box grabbing device 80 comprises:

[0094] The acquisition module 81 is configured to acquire initial three-dimensional point cloud information of a target box.

[0095] The processing module 82 is configured to process the initial three-dimensional point cloud information to obtain an overall two-dimensional mask image of the target box on a preset surface.

[0096] The detection module 83 is configured to perform corner point detection on the overall two-dimensional mask image to obtain a two-dimensional mask image of at least one target box, and the preset shape is the shape of a surface of the target box.

[0097] The determination module 84 is configured to determine the two-dimensional mask image of the at least one target box according to the preset shape and the corner point parameters of the overall two-dimensional mask image.

[0098] The control module 85 is configured to control a grabbing device to grab the corresponding target box according to the two-dimensional mask image of the at least one target box.

[0099] In an optional embodiment, the processing module 82 is specifically configured to: pre-process the initial three-dimensional point cloud information to obtain target three-dimensional point cloud information, the target three-dimensional point cloud information comprising three-dimensional point cloud information of the target box body on the preset surface; and map the target three-dimensional point cloud information into an overall two-dimensional mask image.

[0100] In an optional embodiment, when the processing module 82 maps the target three-dimensional point cloud information into the overall two-dimensional mask image, the processing module 82 is specifically configured to: project the target three-dimensional point cloud information into two-dimensional information; and determine a region corresponding to the two-dimensional information as the overall two-dimensional mask image, each pixel point in the overall two-dimensional mask image being white, and each pixel point having corresponding two-dimensional information.

[0101] In an optional embodiment, when the processing module 82 pre-processes the initial three-dimensional point cloud information to obtain the target three-dimensional point cloud information, the processing module 82 is specifically configured to: adopt a clustering analysis manner to delete three-dimensional point cloud information not belonging to the preset surface in the initial three-dimensional point cloud information to obtain the target three-dimensional point cloud information of the preset surface; and / or remove outliers in the initial three-dimensional point cloud information to obtain the target three-dimensional point cloud information.

[0102] In an optional embodiment, when the processing module 82 adopts the clustering analysis manner to delete the three-dimensional point cloud information not belonging to the preset surface to obtain the target three-dimensional point cloud information of the preset surface, the processing module 82 is specifically configured to: determine normal information of each point in the initial three-dimensional point cloud information; and cluster points belonging to the same normal information in the initial three-dimensional point cloud information to obtain a plurality of clusters; determine a cluster comprising the largest number of points as a target cluster; and determine three-dimensional point cloud information belonging to the target cluster as the target three-dimensional point cloud information of the preset surface.

[0103] In an optional embodiment, the determining module 84 is specifically configured to: segment a connecting line between each corner point of the overall two-dimensional mask image according to a preset shape; and screen out a corner point surrounding shape consistent with the preset shape as a two-dimensional mask image of each target box body.

[0104] In an optional embodiment, the control module 85 is specifically configured to: extract three-dimensional point cloud information of a single target box body from the target three-dimensional point cloud information according to the two-dimensional mask image of the at least one target box body; determine pose information and size information of the target box body according to the three-dimensional point cloud information of the single target box body; and control the grabbing device to grab the corresponding single target box body according to the pose information and the size information.

[0105] In an optional embodiment, the acquisition module 81 is specifically configured to control the industrial three-dimensional camera to collect point clouds of the target box body, and obtain initial three-dimensional point cloud information; wherein the target box body is placed on the conveying device, and the target box body is conveyed by the conveying device into the collection angle of the industrial three-dimensional camera.

[0106] The box body grabbing device provided by the present disclosure can realize single and accurate grabbing of the target box body at one time, and avoid grabbing errors. In addition, since the present disclosure uses a two-dimensional mask image to grab the target box body, the influence of the pattern on the target box body on the grabbing can be avoided.

[0107] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appearing in a specific order are included, but it should be clear that these operations can be executed in the order appearing in the present disclosure or in parallel, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in the present disclosure are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence. In addition, "first" and "second" are different types.

[0108] Figure 9 The structural schematic diagram of the electronic device provided by an example embodiment of the present disclosure is shown in FIG. 9. Figure 9 As shown in FIG. 9, the electronic device 90 includes a processor 91 and a memory 92 connected to the processor 91 in communication, and the memory 92 stores computer execution instructions.

[0109] The processor executes the computer execution instructions stored in the memory to implement the box body grabbing method provided by any of the above method embodiments, and the specific functions and technical effects that can be achieved are not repeated here.

[0110] The present disclosure further provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions. When the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the box body grabbing method provided by any of the above method embodiments.

[0111] The present disclosure further provides a computer program product, and the program product includes a computer program stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to make the electronic device execute the box body grabbing method provided by any of the above method embodiments.

[0112] In several embodiments provided in the present disclosure, it should be understood that the disclosed system and method can be implemented in other manners. For example, the system embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, and electrical, mechanical or other forms.

[0113] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. In actual implementation, a part or all of the units can be selected according to the actual needs to achieve the purposes of the embodiments.

[0114] In addition, each functional unit in the various embodiments of the present disclosure can be integrated in one processing unit, or each unit can exist physically as separated, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware, or in the form of hardware plus software function units.

[0115] The integrated unit implemented in the form of software function units can be stored in a computer readable storage medium. The above software function unit stored in a storage medium includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method of the various embodiments of the present disclosure. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes.

[0116] It can be clearly understood by those skilled in the art that, for the convenience and brevity, only the division of the above functional modules is taken as an example for description. In actual application, the above functions can be completed by different functional modules according to needs, i.e., the internal structure of the system is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0117] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the disclosure be construed as including any paterns of this disclosure that can be derived from the description and illustrations presented herein without departing from the scope and spirit of the disclosure. The specification and examples are exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0118] It should be understood that the present disclosure is not limited to the precise structures as herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A box gripping method, characterized in that, include: Obtain the initial 3D point cloud information of the target box; The initial three-dimensional point cloud information is processed to obtain a two-dimensional mask image of the target box on a preset surface; Corner detection is performed on the overall two-dimensional mask image to obtain the corner parameters of the overall two-dimensional mask image; Based on the preset shape and the corner point parameters of the acquired overall two-dimensional mask image, at least one two-dimensional mask image of the target box is determined, wherein the preset shape at least includes the shape of the grasping surface of the target box; Based on the two-dimensional mask image of at least one of the target boxes, the grasping device is controlled to grasp the corresponding target box.

2. The box gripping method according to claim 1, characterized in that, The process of processing the initial 3D point cloud information to obtain a 2D mask image of the target box on a preset surface includes: The initial three-dimensional point cloud information is preprocessed to obtain the target three-dimensional point cloud information, which includes the three-dimensional point cloud information of the target box on a preset surface. The target 3D point cloud information is mapped to the overall 2D mask image.

3. The box gripping method according to claim 2, characterized in that, The step of mapping the target 3D point cloud information into the overall 2D mask image includes: Project the target three-dimensional point cloud information into two-dimensional information; The region corresponding to the two-dimensional information is determined as the overall two-dimensional mask image, where each pixel in the overall two-dimensional mask image is white and each pixel has corresponding two-dimensional information.

4. The box gripping method according to claim 2, characterized in that, The preprocessing of the initial 3D point cloud information to obtain the target 3D point cloud information includes: Cluster analysis is used to delete 3D point cloud information that does not belong to the preset surface from the initial 3D point cloud information to obtain the target 3D point cloud information of the preset surface. And / or, The outliers in the initial 3D point cloud information are removed to obtain the target 3D point cloud information.

5. The box gripping method according to claim 4, characterized in that, In the case of using cluster analysis, the step of deleting 3D point cloud information that does not belong to the preset surface from the initial 3D point cloud information to obtain the target 3D point cloud information of the preset surface includes: In the initial 3D point cloud information, determine the normal information of each point; In the initial 3D point cloud information, points belonging to the same normal information are clustered to obtain multiple clusters; The cluster containing the most points is identified as the target cluster. The three-dimensional point cloud information belonging to the target cluster is determined as the target three-dimensional point cloud information of the preset surface.

6. The box gripping method according to any one of claims 1 to 5, characterized in that, Based on the preset shape and the corner point parameters of the acquired overall two-dimensional mask image, a two-dimensional mask image of a single target box is obtained, including: According to the preset shape, dividing lines are made between each corner point of the overall two-dimensional mask image; Filter out all corner-surrounding shapes that match the preset shape, and use them as two-dimensional mask images of each target box.

7. The box gripping method according to any one of claims 2 to 5, characterized in that, The step of controlling the grasping device to grasp the corresponding target box based on the two-dimensional mask image of the at least one target box includes: Based on the two-dimensional mask image of the at least one target box, extract the three-dimensional point cloud information of a single target box from the three-dimensional point cloud information of the target; Based on the three-dimensional point cloud information of the single target box, determine the pose information and size information of the target box; Based on the pose information and the size information, the grasping device is controlled to grasp the corresponding single target box.

8. The box gripping method according to any one of claims 1 to 5, characterized in that, The acquisition of initial 3D point cloud information includes: The industrial 3D camera is controlled to collect point cloud data of the target box, and the initial 3D point cloud information is obtained. The target box is placed on a conveying device and is conveyed by the conveying device to the acquisition field of view of the industrial 3D camera.

9. A box gripping device, characterized in that, include: The acquisition module is used to acquire the initial 3D point cloud information of the target box. The processing module is used to process the initial three-dimensional point cloud information to obtain the overall two-dimensional mask image of the target box on the preset surface; The detection module is used to perform corner detection on the overall two-dimensional mask image in order to obtain the corner parameters of the overall two-dimensional mask image; The determining module is used to determine at least one two-dimensional mask image of the target box based on a preset shape and the corner point parameters of the acquired overall two-dimensional mask image; The control module is used to control the grasping device to grasp the corresponding target box based on a two-dimensional mask image of at least one of the target boxes.

10. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the box grasping method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the box grasping method of any one of claims 1 to 8.

Images