Pickup pose information determination method, device, equipment and computer readable medium

Abstract

This disclosure provides embodiments of a method, apparatus, device, and computer-readable medium for determining pickup pose information. One specific implementation of the method includes: generating an edge detection image using a color image and a depth image captured for a source region; performing connected component segmentation on the edge detection image to obtain a connected component segmentation image, wherein the connected component segmentation image includes at least one connected component; classifying the connected components in the connected component segmentation image to obtain a set of pickup connected components; and determining the pickup pose information of the object represented by each pickup connected component in the set of pickup connected components to obtain a set of pickup pose information. This implementation avoids using prior information about the object and improves the accuracy of target detection, thereby enabling more precise determination of pickup pose information.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of computer technology, and more specifically to a method, apparatus, device, and computer-readable medium for determining pose information. Background Technology

[0002] Automated depalletizing and palletizing refers to a technology where automated depalletizing and palletizing equipment, based on vision guidance, performs target detection on items in the source area according to received depalletizing and palletizing tasks, and then picks up the corresponding number of items in the source area and places them into a designated destination area. Currently, mathematical calculation methods, traditional computer vision algorithms, or deep learning algorithms are often used to perform target detection on items in the source area.

[0003] However, when using the above methods for target detection, the following technical problems often arise:

[0004] First, mathematical solution methods and traditional computer vision algorithms rely on prior information about the items. However, items are updated and replaced quickly, requiring frequent collection of prior information. Moreover, the collection of prior information is time-consuming, which reduces the adaptability of automatic depalletizing and palletizing methods.

[0005] Second, the accuracy of object detection in deep learning methods depends on the amount of data used to train the model. However, in automated depalletizing and palletizing scenarios, it is difficult to collect large-scale data. The texture and shadows of objects in the image can also affect the accuracy of object detection, resulting in inaccurate picking pose estimation and reducing the success rate of automated depalletizing and palletizing. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure provide methods, apparatus, devices, and computer-readable media for determining pose information to address one or more of the technical problems mentioned in the background section above.

[0008] In a first aspect, some embodiments of this disclosure provide a method for determining pickup pose information. The method includes: generating an edge detection image using a color image and a depth image captured for a source region; performing connected component segmentation processing on the edge detection image to obtain a connected component segmentation image, wherein the connected component segmentation image includes at least one connected component; classifying the connected components in the connected component segmentation image to obtain a pickup connected component set; and determining the pickup pose information of the item represented by each pickup connected component in the pickup connected component set to obtain a pickup pose information set.

[0009] Optionally, the above method further includes: sending the above-mentioned set of pickup pose information to an automatic depalletizing and palletizing device.

[0010] Optionally, the above-mentioned method of generating an edge detection image using a color image and a depth image captured for the source region includes: extracting a region of interest from the depth image; applying a mask to the color image using the region of interest to obtain a mask image; and performing edge detection processing on the mask image to obtain an edge detection image.

[0011] Optionally, the above-mentioned classification processing of connected components in the connected component segmentation image to obtain a set of picked connected components includes: determining a reference area value based on the area value of each connected component in the connected component segmentation image; classifying each connected component in the connected component segmentation image using the reference area value to obtain a target set of connected components, a set of connected components to be split, and a set of connected components to be spliced; and performing clustering processing on the target set of connected components, the set of connected components to be split, and the set of connected components to be spliced ​​to obtain a set of picked connected components.

[0012] Optionally, the above-mentioned clustering process of the target connected component set, the connected component set to be split, and the connected component set to be joined to obtain the picking connected component set includes: clustering each connected component to be split in the connected component set to be split based on the reference area value to obtain a first clustered connected component set; adding the first clustered connected component in the first clustered connected component set that meets a first preset condition as a target connected component to the target connected component set; and adding the first clustered connected component in the first clustered connected component set that meets a second preset condition as a connected component to the connected component set to be joined.

[0013] Optionally, the above-mentioned clustering process of the target connected component set, the connected component set to be split, and the connected component set to be spliced ​​to obtain the picking connected component set further includes: performing clustering process on each connected component to be spliced ​​in the connected component set to be spliced ​​to obtain a second clustered connected component set; adding the second clustered connected component in the second clustered connected component set that satisfies the first preset condition as the target connected component to the target connected component set; and determining the target connected component set as the picking connected component set.

[0014] Optionally, the extraction of the region of interest (ROI) from the depth image includes: generating a plane equation for the source region using the location information of the source region and the depth image; selecting pixels located within the source region in the depth image as target pixels using the depth information of pixels in the depth image and the plane equation of the source region, thus obtaining a set of target pixels; converting each target pixel in the target pixel set into point cloud data, thus obtaining a set of point cloud data; performing plane fitting processing on the point cloud data set to obtain a fitted plane equation; selecting point cloud data that satisfies a third preset condition from the point cloud data set as target point cloud data according to the fitted plane equation, thus obtaining a set of target point cloud data; converting the target point cloud data in the target point cloud data set to a two-dimensional coordinate system using the intrinsic parameters of the camera that captured the depth image, thus obtaining a set of two-dimensional coordinates; and generating the ROI based on the two-dimensional coordinate set.

[0015] Optionally, the above-mentioned clustering process for each connected component to be split in the set of connected components to be split to obtain a first clustered connected component set includes: determining the number of split clusters based on the ratio of the area of ​​the connected component to be split to the reference area value; and splitting the connected component to be split based on the number of split clusters to obtain each first clustered connected component.

[0016] Optionally, determining the reference area value based on the area values ​​of each connected component in the connected component segmentation image includes: determining the mode of the area values ​​of each connected component in the connected component segmentation image as the reference area value.

[0017] Optionally, generating the region of interest based on the above two-dimensional coordinate set includes: determining the smallest bounding polygon of the point set corresponding to the above two-dimensional coordinate set as the region of interest.

[0018] Secondly, some embodiments of this disclosure provide a pickup pose information determination apparatus, the apparatus comprising: a generation unit configured to generate an edge detection image using a color image and a depth image captured for a source region; a segmentation unit configured to perform connected component segmentation processing on the edge detection image to obtain a connected component segmentation image, wherein the connected component segmentation image includes at least one connected component; a classification unit configured to perform classification processing on the connected components in the connected component segmentation image to obtain a pickup connected component set; and a determination unit configured to determine the pickup pose information of an item represented by each pickup connected component in the pickup connected component set to obtain a pickup pose information set.

[0019] Optionally, the above-mentioned device further includes a transmitting unit configured to transmit the above-mentioned pickup pose information set to an automatic depalletizing device.

[0020] Optionally, the generation unit includes an extraction subunit, a masking subunit, and an edge detection subunit. The extraction subunit is configured to extract the region of interest (ROI) from the depth image; the masking subunit is configured to apply a mask to the color image using the ROI to obtain a mask image; and the edge detection subunit is configured to perform edge detection on the mask image to obtain an edge detection image.

[0021] Optionally, the classification unit includes a first determining subunit and a classification subunit. The first determining subunit is configured to determine a reference area value based on the area values ​​of each connected component in the connected component segmentation image; the classification subunit is configured to classify each connected component in the connected component segmentation image using the reference area value to obtain a target connected component set, a set of connected components to be split, and a set of connected components to be spliced; the clustering subunit is configured to perform clustering processing on the target connected component set, the set of connected components to be split, and the set of connected components to be spliced ​​to obtain a picked connected component set.

[0022] Optionally, the clustering subunit further includes: a first clustering module, a first addition module, and a second addition module. The first clustering module is configured to perform clustering processing on each connected component to be split in the set of connected components to be split based on the reference area value, resulting in a first clustered connected component set; the first addition module is configured to add the first clustered connected components in the first clustered connected component set that satisfy a first preset condition as target connected components to the target connected component set; and the second addition module is configured to add the first clustered connected components in the first clustered connected component set that satisfy a second preset condition as connected components to be spliced ​​to the set of connected components to be spliced.

[0023] Optionally, the clustering subunit further includes: a second clustering module, a third adding module, and a second determining module. The second clustering module is configured to perform clustering processing on each connected component to be joined in the set of connected components to be joined, obtaining a second clustered connected component set; the third adding module is configured to add the second clustered connected components in the second clustered connected component set that satisfy the first preset condition as target connected components to the target connected component set; and the second determining module is configured to determine the target connected component set as the picking connected component set.

[0024] Optionally, the above extraction sub-unit includes: a first generation module, a first selection module, a first transformation module, a plane fitting module, a second selection module, a second transformation module, and a second generation module. The system comprises the following modules: a first generation module, configured to generate a source region plane equation using the location information of the source region and the depth image; a first selection module, configured to select pixels located within the source region in the depth image as target pixels using the depth information of pixels in the depth image and the source region plane equation, thereby obtaining a target pixel set; a first conversion module, configured to convert each target pixel in the target pixel set into point cloud data, thereby obtaining a point cloud data set; a plane fitting module, configured to perform plane fitting processing on the point cloud data set, thereby obtaining a fitted plane equation; a second selection module, configured to select point cloud data satisfying a third preset condition from the point cloud data set according to the fitted plane equation, thereby obtaining a target point cloud data set; a second conversion module, configured to convert the target point cloud data in the target point cloud data set to a two-dimensional coordinate system using the intrinsic parameters of the camera that captured the depth image, thereby obtaining a two-dimensional coordinate set; and a second generation module, configured to generate a region of interest based on the two-dimensional coordinate set.

[0025] Optionally, the first clustering module includes a determining submodule and a splitting submodule. The determining submodule is configured to determine the number of split clusters based on the ratio of the area of ​​the connected component to be split to the reference area value; the splitting submodule is configured to split the connected component to be split based on the number of split clusters to obtain each first cluster connected component.

[0026] Optionally, the first determining subunit is further configured to determine the mode of the area values ​​of each connected component in the connected component segmentation image as the reference area value.

[0027] Optionally, the first conversion module described above is further configured to determine the minimum bounding polygon of the point set corresponding to the two-dimensional coordinate set as the region of interest.

[0028] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0029] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0030] The above-described embodiments of this disclosure have the following beneficial effects: The method for determining pickup pose information according to some embodiments of this disclosure can avoid using prior information about the item and improve the accuracy of target detection. Furthermore, it can more accurately determine the pickup pose information and improve the success rate of automatic depalletizing. Specifically, the reason for the low accuracy of target detection and the difficulty in accurately determining the pickup pose information is that mathematical calculation methods and traditional computer vision algorithms rely on prior information about the item, while the accuracy of target detection in deep learning methods depends on the amount of data used for model training. Based on this, the method for determining pickup pose information according to some embodiments of this disclosure uses color images and depth images captured for the source region to generate an edge detection image. This yields a preliminary edge detection result. Next, the connected components in the edge detection image are classified to obtain a set of picked connected components. Further processing is performed on the edge detection image, and the picked connected components in the obtained set are used as the final, more accurate target detection result. This results in higher accuracy of the pickup pose information generated based on the picked connected components in the set. Because prior information about the items is not used in the process of determining the pickup pose information, the technical problem of poor adaptability of the automatic depalletizing method caused by the frequent collection of prior information and long collection time is avoided. Furthermore, based on the obtained target detection result—the edge detection image—connected components are further classified, thereby improving the accuracy of the target detection result. This allows for more precise determination of the pickup pose information and increases the success rate of automatic depalletizing. Attached Figure Description

[0031] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0032] Figure 1This is a schematic diagram illustrating an application scenario of the method for determining pose information according to some embodiments of this disclosure;

[0033] Figure 2 This is a flowchart of some embodiments of the method for determining pose information according to this disclosure;

[0034] Figure 3 This is a flowchart of some other embodiments of the method for determining pose information according to this disclosure;

[0035] Figure 4 This is a schematic diagram illustrating the determination of a set of connected components in some other embodiments of the method for determining pickup pose information according to this disclosure;

[0036] Figure 5 This is a schematic diagram of the structure of some embodiments of the device for determining pose information in this disclosure;

[0037] Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0038] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0039] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0040] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0041] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0042] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0043] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0044] Figure 1 This is a schematic diagram illustrating an application scenario of the method for determining pose information based on some embodiments of this disclosure.

[0045] exist Figure 1 In the application scenario, firstly, the computing device 101 can generate an edge detection image 105 using a color image 103 and a depth image 104 captured for the source region 102. Next, the computing device 101 can perform connected component segmentation processing on the edge detection image 105 to obtain a connected component segmentation image 106, wherein the connected component segmentation image 106 includes at least one connected component. Then, the computing device 101 can classify the connected components in the connected component segmentation image 106 to obtain a set of picked connected components 107. Finally, the computing device 101 can determine the picking pose information of the item represented by each picked connected component in the set of picked connected components 107, obtaining a set of picking pose information 108.

[0046] It should be noted that the aforementioned computing device 101 can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed within the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0047] It should be understood that Figure 1 The number of computing devices shown is merely illustrative. Any number of computing devices can be used depending on implementation needs.

[0048] Continue to refer to Figure 2 The flowchart 200 illustrates some embodiments of the method for determining pickup pose information according to the present disclosure. The method for determining pickup pose information includes the following steps:

[0049] Step 201: Generate an edge detection image using the color image and depth image captured for the source region.

[0050] In some embodiments, the aforementioned color image and depth image can be images captured simultaneously. The aforementioned color image can be a color image such as an RGB (red, green, blue) image captured by a common 2D camera. The aforementioned depth image can be captured by a 3D camera, or it can be obtained by converting an image captured by the aforementioned 2D camera. The aforementioned source region can be an area where an object that needs to be moved or transferred is placed. The execution entity of the pose information determination method (e.g., Figure 1The computing device 101 shown generates an edge detection image using a color image and a depth image captured for the source region, which may include the following steps:

[0051] The first step is to convert the above color image to the YCbCr color space to obtain a YCbCr image.

[0052] The second step involves generating Y gradient images, Cb gradient images, and Cr gradient images using the luminance component Y, color components Cb, and Cr of the aforementioned YCbCr image. An 8-directional circular edge detection operator can be used to generate each gradient image.

[0053] The third step is to generate a depth gradient map using the aforementioned depth image. This involves normalizing the depth values ​​of each pixel in the depth image and then using the normalized depth values ​​to generate the depth gradient map. The aforementioned 8-direction circular edge detection operator can be used to generate this depth gradient map.

[0054] The fourth step involves fusing the Y gradient image, Cb gradient image, Cr gradient image, and depth gradient image to obtain the target gradient image. This fusion can be achieved using a linear weighting method.

[0055] The fifth step is to take the maximum value of the 8-directional gradient of the target gradient map as the edge detection result to obtain the edge detection image.

[0056] Step 202: Perform connected component segmentation on the edge detection image to obtain a connected component segmentation image.

[0057] In some embodiments, the execution entity may perform connected component segmentation on the edge detection image to obtain a connected component segmented image. The connected component segmented image includes at least one connected component. Connected component segmentation algorithms can be used to perform connected component segmentation on the edge detection image. These algorithms include, but are not limited to, at least one of the following: Two-Pass method, Seed-Filling method, etc.

[0058] Step 203: Classify the connected components in the connected component segmentation image to obtain the picked connected component set.

[0059] In some embodiments, the execution entity performs classification processing on the connected components in the connected component segmentation image to obtain a set of picked connected components, which may include the following steps:

[0060] The first step is to perform image dilation processing on the edges of each connected component in the above connected component segmentation image to obtain a dilated connected component segmentation image.

[0061] The second step involves identifying the dilated connected components in the image segmented by the above-mentioned dilated connected components whose ratio of actual area to preset area value satisfies a preset condition as the picked connected components, thus obtaining a set of picked connected components. The preset condition can be that the ratio is within a preset range. In practice, the preset range can be set according to the actual application. For example, the preset range could be [0.8, 1.2]. The preset area value can be the actual area value of each item in the source region.

[0062] The actual area of ​​the aforementioned expanded connected region can be determined through the following sub-steps:

[0063] The first sub-step involves determining the average depth value of each pixel in the region corresponding to the dilated connected component in the depth image as the distance between the object represented by the dilated connected component and the camera that captured the depth image.

[0064] The second sub-step involves using the aforementioned distance and the intrinsic parameters of the camera that captured the aforementioned depth image to convert the area of ​​the aforementioned dilated connected region in the image into the actual area.

[0065] Step 204: Determine the pickup pose information of the item represented by each pickup connected component in the pickup connected component set, and obtain the pickup pose information set.

[0066] In some embodiments, the execution entity determines the pickup pose information of the item represented by each pickup connected component in the pickup connected component set, which may include the following steps:

[0067] The first step is to use the intrinsic and extrinsic parameters of the camera that captured the above color images to convert the two-dimensional coordinates within the picked connected domain into three-dimensional coordinates, thus obtaining a set of three-dimensional coordinates.

[0068] The second step is to perform plane fitting on the above three-dimensional coordinate set to obtain the object fitting plane equation.

[0069] The third step is to use the intrinsic and extrinsic parameters of the camera used to capture the color image to convert the coordinates of the two-dimensional center point of the picked connected domain into the coordinates of the three-dimensional center point.

[0070] The fourth step is to use the normal vector of the fitted plane equation of the above-mentioned object and the coordinates of the above-mentioned three-dimensional center point as the picking pose information.

[0071] In some optional implementations of certain embodiments, the aforementioned execution entity may also send the aforementioned set of pickup pose information to an automatic depalletizing and palletizing device.

[0072] The above-described embodiments of this disclosure have the following beneficial effects: The method for determining pickup pose information according to some embodiments of this disclosure can avoid using prior information about the item and improve the accuracy of target detection. Furthermore, it can more accurately determine the pickup pose information and improve the success rate of automatic depalletizing. Specifically, the reason for the low accuracy of target detection and the difficulty in accurately determining the pickup pose information is that mathematical calculation methods and traditional computer vision algorithms rely on prior information about the item, while the accuracy of target detection in deep learning methods depends on the amount of data used for model training. Based on this, the method for determining pickup pose information according to some embodiments of this disclosure uses color images and depth images captured for the source region to generate an edge detection image. This yields a preliminary edge detection result. Next, the connected components in the edge detection image are classified to obtain a set of picked connected components. Further processing is performed on the edge detection image, and the picked connected components in the obtained set are used as the final, more accurate target detection result. This results in higher accuracy of the pickup pose information generated based on the picked connected components in the set. Because prior information about the items is not used in the process of determining the pickup pose information, the technical problem of poor adaptability of the automatic depalletizing method caused by the frequent collection of prior information and long collection time is avoided. Furthermore, based on the obtained target detection result—the edge detection image—connected components are further classified, thereby improving the accuracy of the target detection result. This allows for more precise determination of the pickup pose information and increases the success rate of automatic depalletizing.

[0073] Further reference Figure 3 The diagram illustrates a flow 300 of another embodiment of the method for determining pose information. Flow 300 of this method includes the following steps:

[0074] Step 301: Generate an edge detection image using the color image and depth image captured for the source region.

[0075] In some embodiments, the color image and the depth image may be images captured simultaneously. The color image may be an RGB image captured by a common 2D camera, etc. The depth image may be captured by a 3D camera, or it may be obtained by converting an image captured by a 2D camera. The source region may be an area where an object that needs to be moved or transferred is placed. The execution entity of the pose information determination method (e.g., Figure 1 The computing device 101 shown generates an edge detection image using a color image and a depth image captured for the source region, which may include the following steps:

[0076] Step 3011: Extract the region of interest from the depth image.

[0077] In some embodiments, the extraction of the region of interest from the depth image by the execution entity may include the following steps:

[0078] The first step is to generate a planar equation for the source region using the location information and depth image of the source region. The location information of the source region may include the three-dimensional coordinates of each corner point of the source region. This location information may be pre-acquired and stored. The extent of the source region can be determined using the three-dimensional coordinates of each corner point of the source region. The planar equation for the source region can be generated using the three-dimensional coordinates of each corner point of the source region and the depth information from the depth image.

[0079] The second step involves using the depth information of pixels in the depth image and the plane equation of the source region to select pixels located within the source region from the depth image as target pixels, thus obtaining a set of target pixels. Specifically, the distance between each pixel and the plane equation of the source region can be determined based on the depth information of each pixel in the depth image. If the distance is non-negative, then the pixel is determined to be located within the source region.

[0080] The third step is to convert each target pixel in the above target pixel set into point cloud data to obtain a point cloud data set.

[0081] The fourth step is to perform plane fitting on the above point cloud dataset to obtain the fitted plane equation.

[0082] Fifth, based on the fitted plane equation described above, select point cloud data that satisfy the third preset condition from the aforementioned point cloud dataset as target point cloud data, thus obtaining the target point cloud dataset. The third preset condition can be that the distance between the points represented by the point cloud data and the fitted plane equation is greater than a target distance value. The target distance value can be the arithmetic mean of the distances between the points represented by each point cloud data point in the point cloud dataset and the fitted plane equation.

[0083] Step 6: Using the intrinsic parameters of the camera that captured the aforementioned depth image, the target point cloud data in the aforementioned target point cloud dataset is transformed into a two-dimensional coordinate system to obtain a two-dimensional coordinate set.

[0084] Step 7: Generate the region of interest based on the above two-dimensional coordinate set.

[0085] Optionally, the aforementioned execution entity may determine the minimum bounding polygon of the point set corresponding to the aforementioned two-dimensional coordinate set as the region of interest.

[0086] Step 3012: Perform masking on the color image using the region of interest to obtain a mask image.

[0087] The masking process described above can involve setting the pixel values ​​of pixels located outside the region of interest in the color image to preset values. In practice, these preset values ​​can be set according to actual conditions and are not limited here. For example, the preset value could be 0.

[0088] Step 3013: Perform edge detection processing on the mask image to obtain an edge detection image.

[0089] Edge detection operators can be used to perform edge detection processing on the above-mentioned mask image. These edge detection operators may include, but are not limited to, the Sobel operator, the Prewitt operator, and the Roberts operator.

[0090] Step 302: Perform connected component segmentation on the edge detection image to obtain a connected component segmentation image.

[0091] In some embodiments, the specific implementation of step 302 and its resulting technical effects can be found in [reference needed]. Figure 2 Step 202 in the corresponding embodiments will not be repeated here.

[0092] Step 303: Classify the connected components in the connected component segmentation image to obtain the picked connected component set.

[0093] In some embodiments, the execution entity performs classification processing on the connected components in the connected component segmentation image to obtain a set of picked connected components, which may include the following steps:

[0094] Step 3031: Determine the reference area value based on the area values ​​of each connected component in the connected component segmentation image.

[0095] The mode of the area values ​​of each connected component in the above connected component segmentation image can be determined as the reference area value.

[0096] Step 3032: Using the reference area value, classify each connected component in the connected component segmentation image to obtain the target connected component set, the connected component set to be split, and the connected component set to be spliced.

[0097] First, the ratio of the area value of each connected component in the segmented image to the aforementioned reference area value can be determined, resulting in a set of area ratios. Then, connected components with an area ratio greater than or equal to a first area value and less than a second area value are selected from the segmented image as the connected components to be spliced, resulting in a set of connected components to be spliced. Next, connected components with an area ratio greater than or equal to a second area value and less than a third area value are selected from the segmented image as the target connected components, resulting in a set of target connected components. Finally, connected components with an area ratio greater than or equal to a third area value and less than a fourth area value are selected from the segmented image as the connected components to be split, resulting in a set of connected components to be split.

[0098] In practice, the first, second, and third area values ​​can be set according to the actual application, and are not limited here. For example, the first area value can be 0.2, the second area value can be 0.8, the third area value can be 1.2, and the third area value can be 10.

[0099] Step 3033: Perform clustering on the target connected component set, the connected component set to be split, and the connected component set to be joined to obtain the picked connected component set.

[0100] In some embodiments, the execution entity performs clustering processing on the target connected component set, the connected component set to be split, and the connected component set to be joined to obtain a picked connected component set, which may include the following steps:

[0101] The first step involves clustering each connected component in the set of connected components to be split based on the aforementioned reference area value, resulting in a first clustered set of connected components. Clustering algorithms can be used to cluster these connected components. These algorithms may include, but are not limited to, K-Means clustering and RCF (Richer Convolutional Features) networks.

[0102] Optionally, the execution entity may perform clustering processing on each connected component to be split in the set of connected components to be split based on the reference area value to obtain a first clustered connected component set, which may include the following sub-steps:

[0103] The first sub-step involves determining the number of split clusters based on the ratio of the area of ​​the connected component to be split to the reference area value. This ratio can be rounded to obtain the final number of split clusters.

[0104] The second sub-step involves splitting the connected components to be split based on the aforementioned number of split clusters, resulting in each first cluster connected component. The aforementioned clustering algorithm can be used to split the connected components to be split.

[0105] The second step is to add the first clustered connected component that satisfies the first preset condition from the first clustered connected component set to the target connected component set. The first preset condition may be that the ratio of the area of ​​the first clustered connected component to the reference area value is greater than or equal to the second area value and less than the third area value.

[0106] The third step is to add the first clustered connected component that meets the second preset condition from the first clustered connected component set as the connected component to be spliced ​​into the set of connected components to be spliced. The second preset condition may be that the ratio of the area of ​​the first clustered connected component to the reference area value is greater than or equal to the first area value and less than the second area value.

[0107] Optionally, the execution entity may further perform clustering processing on the target connected component set, the connected component set to be split, and the connected component set to be joined to obtain the picked connected component set, and may also include the following steps:

[0108] The first step is to cluster the connected components in the aforementioned set of connected components to be joined, resulting in a second clustered set of connected components. The clustering algorithm described above can be used to cluster the connected components in the aforementioned set of connected components to be joined.

[0109] The second step is to add the second clustered connected domains that satisfy the first preset condition from the second clustered connected domain set to the target connected domain set.

[0110] The third step is to determine the target connected component set as the picking connected component set.

[0111] As an example, refer to Figure 4 First, the connected components to be joined in the set 401 can be clustered to obtain a second clustered connected component set 402. Then, the second clustered connected components in the second clustered connected component set 402 that satisfy the first preset condition 403 can be added to the target connected component set 404 as target connected components. Finally, the target connected component set 404 can be determined as the picking connected component set 405.

[0112] Therefore, clustering can be used to further divide and process the connected components in the edge detection image, thereby achieving more accurate target recognition.

[0113] Step 304: Determine the pickup pose information of the item represented by each pickup connected component in the pickup connected component set, and obtain the pickup pose information set.

[0114] In some embodiments, the specific implementation of step 304 and its resulting technical effects can be found in [reference needed]. Figure 2 Step 204 in the corresponding embodiments will not be repeated here.

[0115] from Figure 3 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 3 The flowchart 300 of the method for determining the pickup pose information in some corresponding embodiments reflects the use of a clustering algorithm to further cluster the preliminary edge detection result—the edge detection image. Therefore, the schemes described in these embodiments can more accurately divide the connected components in the edge detection image through clustering. This facilitates the determination of the pickup pose based on the accurately divided connected components.

[0116] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a device for determining and picking up pose information. These device embodiments are similar to... Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0117] like Figure 5 As shown, a pickup pose information determination device 500 in some embodiments includes: a generation unit 501, a segmentation unit 502, a classification unit 503, and a determination unit 504. The generation unit 501 is configured to generate an edge detection image using a color image and a depth image captured for a source region; the segmentation unit 502 is configured to perform connected component segmentation processing on the edge detection image to obtain a connected component segmentation image, wherein the connected component segmentation image includes at least one connected component; the classification unit 503 is configured to classify the connected components in the connected component segmentation image to obtain a pickup connected component set; and the determination unit 504 is configured to determine the pickup pose information of the item represented by each pickup connected component in the pickup connected component set, thereby obtaining a pickup pose information set.

[0118] In some alternative implementations of the embodiments, the picking pose information determination device 500 further includes a sending unit configured to send the picking pose information set to an automatic depalletizing device.

[0119] In some optional implementations of embodiments, the generation unit 501 includes an extraction subunit, a masking subunit, and an edge detection subunit. The extraction subunit is configured to extract a region of interest (ROI) from the depth image; the masking subunit is configured to perform masking processing on the color image using the ROI to obtain a mask image; and the edge detection subunit is configured to perform edge detection processing on the mask image to obtain an edge detection image.

[0120] In some optional implementations of the embodiments, the classification unit 503 includes a first determining subunit and a classification subunit. The first determining subunit is configured to determine a reference area value based on the area values ​​of each connected component in the connected component segmentation image; the classification subunit is configured to classify each connected component in the connected component segmentation image using the reference area value to obtain a target connected component set, a set of connected components to be split, and a set of connected components to be spliced; the clustering subunit is configured to perform clustering processing on the target connected component set, the set of connected components to be split, and the set of connected components to be spliced ​​to obtain a picked connected component set.

[0121] In some optional implementations of embodiments, the clustering subunit further includes: a first clustering module, a first addition module, and a second addition module. The first clustering module is configured to perform clustering processing on each connected component to be split in the set of connected components to be split based on the reference area value, resulting in a first clustered connected component set. The first addition module is configured to add the first clustered connected components in the first clustered connected component set that satisfy a first preset condition as target connected components to the target connected component set. The second addition module is configured to add the first clustered connected components in the first clustered connected component set that satisfy a second preset condition as connected components to be spliced ​​into the set of connected components to be spliced.

[0122] In some optional implementations of embodiments, the clustering subunit further includes: a second clustering module, a third adding module, and a second determining module. The second clustering module is configured to perform clustering processing on each connected component to be joined in the set of connected components to be joined, obtaining a second clustered connected component set; the third adding module is configured to add the second clustered connected components in the second clustered connected component set that satisfy the first preset condition as target connected components to the target connected component set; and the second determining module is configured to determine the target connected component set as the picking connected component set.

[0123] In some optional implementations of the embodiments, the above extraction subunit includes: a first generation module, a first selection module, a first transformation module, a plane fitting module, a second selection module, a second transformation module, and a second generation module. The system comprises the following modules: a first generation module, configured to generate a source region plane equation using the location information of the source region and the depth image; a first selection module, configured to select pixels located within the source region in the depth image as target pixels using the depth information of pixels in the depth image and the source region plane equation, thereby obtaining a target pixel set; a first conversion module, configured to convert each target pixel in the target pixel set into point cloud data, thereby obtaining a point cloud data set; a plane fitting module, configured to perform plane fitting processing on the point cloud data set, thereby obtaining a fitted plane equation; a second selection module, configured to select point cloud data satisfying a third preset condition from the point cloud data set according to the fitted plane equation, thereby obtaining a target point cloud data set; a second conversion module, configured to convert the target point cloud data in the target point cloud data set to a two-dimensional coordinate system using the intrinsic parameters of the camera that captured the depth image, thereby obtaining a two-dimensional coordinate set; and a second generation module, configured to generate a region of interest based on the two-dimensional coordinate set.

[0124] In some optional implementations of the embodiments, the first clustering module includes a determining submodule and a splitting submodule. The determining submodule is configured to determine the number of split clusters based on the ratio of the area of ​​the connected component to be split to the reference area value; the splitting submodule is configured to split the connected component to be split based on the number of split clusters to obtain each first cluster connected component.

[0125] In some alternative implementations of the embodiments, the first determining subunit is further configured to determine the mode of the area values ​​of each connected component in the connected component segmentation image as a reference area value.

[0126] In some alternative implementations of the embodiments, the first conversion module is further configured to determine the minimum bounding polygon of the point set corresponding to the two-dimensional coordinate set as the region of interest.

[0127] It is understandable that the units described in the device 500 are related to the reference. Figure 2 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 500 and the units contained therein, and will not be repeated here.

[0128] The following is for reference. Figure 6 It shows a schematic diagram of the structure of an electronic device 600 suitable for implementing some embodiments of the present disclosure. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0129] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0130] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0131] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0132] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0133] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0134] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: generate an edge detection image using a color image and a depth image captured for a source region; perform connected component segmentation processing on the edge detection image to obtain a connected component segmentation image, wherein the connected component segmentation image includes at least one connected component; classify the connected components in the connected component segmentation image to obtain a set of picked connected components; and determine the picking pose information of the item represented by each picked connected component in the set of picked connected components to obtain a set of picking pose information.

[0135] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0137] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a generation unit, a segmentation unit, a classification unit, and a determination unit. The names of these units do not necessarily limit the specific unit; for example, the generation unit may also be described as an "edge detection image generation unit."

[0138] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

Claims

1. A method for determining pose information, comprising: An edge detection image is generated using color and depth images captured of the source region; The edge detection image is subjected to connected component segmentation processing to obtain a connected component segmentation image, wherein the connected component segmentation image includes at least one connected component; The connected components in the segmented image are classified to obtain a set of picked connected components, including: performing image dilation processing on the edges of each connected component in the segmented image to obtain a dilated connected component segmentation image; and determining the dilated connected components in the dilated connected component segmentation image whose ratio of actual area to preset area value satisfies a preset condition as picked connected components to obtain a set of picked connected components. Determining the pickup pose information of the item represented by each pickup connected component in the pickup connected component set to obtain a pickup pose information set includes: using the intrinsic and extrinsic parameters of the camera that captured the color image to convert each two-dimensional coordinate in the pickup connected component into three-dimensional coordinates to obtain a three-dimensional coordinate set; performing plane fitting processing on the three-dimensional coordinate set to obtain the item fitting plane equation; using the intrinsic and extrinsic parameters of the camera that captured the color image to convert the two-dimensional center point coordinates of the pickup connected component into three-dimensional center point coordinates; and using the normal vector of the item fitting plane equation and the three-dimensional center point coordinates as pickup pose information.

2. The method according to claim 1, wherein, The method further includes: The set of picked-up pose information is sent to the automatic depalletizing and palletizing equipment.

3. The method according to claim 1, wherein, The step of generating an edge detection image using color and depth images captured for the source region includes: Extract the region of interest from the depth image; The color image is masked using the region of interest to obtain a masked image; Edge detection processing is performed on the mask image to obtain an edge-detected image.

4. The method according to claim 1, wherein, The step of classifying the connected components in the segmented image to obtain a set of picked connected components includes: A reference area value is determined based on the area value of each connected component in the segmented image. Using the reference area value, each connected component in the connected component segmentation image is classified to obtain a target connected component set, a set of connected components to be split, and a set of connected components to be spliced. Clustering is performed on the target connected component set, the connected component set to be split, and the connected component set to be joined to obtain the picked connected component set.

5. The method according to claim 4, wherein, The step of clustering the target connected component set, the connected component set to be split, and the connected component set to be joined to obtain the picked connected component set includes: Based on the reference area value, each connected component to be split in the set of connected components to be split is clustered to obtain a first clustered set of connected components. The first clustered connected component that satisfies the first preset condition in the first clustered connected component set is added to the target connected component set as the target connected component; The first clustered connected component that meets the second preset condition in the first clustered connected component set is added to the connected component set to be spliced.

6. The method according to claim 5, wherein, The step of clustering the target connected component set, the connected component set to be split, and the connected component set to be joined to obtain the picked connected component set further includes: Clustering is performed on each connected component in the set of connected components to be spliced ​​to obtain a second clustered set of connected components; The second clustered connected component that satisfies the first preset condition is added to the target connected component set as the target connected component; The target connected component set is determined as the picking connected component set.

7. The method according to claim 3, wherein, Extracting the region of interest from the depth image includes: Using the location information of the source region and the depth image, a plane equation for the source region is generated; Using the depth information of pixels in the depth image and the plane equation of the source region, pixels located in the source region of the depth image are selected as target pixels to obtain a set of target pixels; Each target pixel in the target pixel set is converted into point cloud data to obtain a point cloud data set; The point cloud dataset is subjected to plane fitting processing to obtain the fitted plane equation; Based on the fitted plane equation, select point cloud data that meets the third preset condition from the point cloud data set as target point cloud data to obtain the target point cloud data set; Using the intrinsic parameters of the camera that captured the depth image, the target point cloud data in the target point cloud dataset is transformed into a two-dimensional coordinate system to obtain a two-dimensional coordinate set; The region of interest is generated based on the set of two-dimensional coordinates.

8. The method according to claim 5, wherein, The step of clustering each connected component in the set of connected components to be split to obtain a first clustered connected component set includes: The number of clusters to be split is determined based on the ratio of the area of ​​the connected component to be split to the reference area value; Based on the number of split clusters, the connected component to be split is split to obtain each first cluster connected component.

9. The method according to claim 4, wherein, The step of determining a reference area value based on the area values ​​of each connected component in the segmented image includes: The mode of the area values ​​of each connected component in the connected component segmentation image is determined as the reference area value.

10. The method according to claim 7, wherein, The step of generating the region of interest based on the set of two-dimensional coordinates includes: The smallest bounding polygon of the point set corresponding to the two-dimensional coordinate set is determined as the region of interest.

11. A device for picking up and determining pose information, comprising: The generation unit is configured to generate an edge detection image using a color image and a depth image captured for the source region; A segmentation unit is configured to perform connected component segmentation processing on the edge detection image to obtain a connected component segmentation image, wherein the connected component segmentation image includes at least one connected component. The classification unit is configured to classify the connected components in the connected component segmentation image to obtain a set of picked connected components, including: performing image dilation processing on the edges of each connected component in the connected component segmentation image to obtain a dilated connected component segmentation image; and determining the dilated connected components in the dilated connected component segmentation image whose ratio of actual area to preset area value satisfies a preset condition as picked connected components to obtain a set of picked connected components. The determining unit is configured to determine the pickup pose information of the item represented by each pickup connected component in the pickup connected component set, thereby obtaining a pickup pose information set, including: converting each two-dimensional coordinate in the pickup connected component into three-dimensional coordinates using the intrinsic and extrinsic parameters of the camera that captured the color image, thereby obtaining a three-dimensional coordinate set; performing plane fitting processing on the three-dimensional coordinate set to obtain an item fitting plane equation; converting the two-dimensional center point coordinates of the pickup connected component into three-dimensional center point coordinates using the intrinsic and extrinsic parameters of the camera that captured the color image; and using the normal vector of the item fitting plane equation and the three-dimensional center point coordinates as pickup pose information.

12. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

13. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

Images