Grabbing method and device based on image segmentation result and electronic equipment

Abstract

The invention provides a grabbing method and device based on an image segmentation result and electronic equipment, and the method comprises the steps: extracting a foreground region of a mask image containing a bearing object and a plurality of grabbing targets, carrying out the segmentation processing of the foreground mask image, and obtaining a sub-image containing each grabbing target; dividing a minimum circumscribed rotating rectangle of a captured target in the sub-images, determining a main shaft based on the minimum circumscribed rotating rectangle, performing pixel point projection on the captured target based on the main shaft, and determining a captured point based on density information of projection points. And on the basis of the grabbing points of all the grabbing targets, candidate grabbing targets without collisions are screened out, the multiple screened candidate grabbing targets are sorted to obtain a grabbing sequence, and the mechanical arm is controlled to execute grabbing according to the grabbing sequence and on the basis of the grabbing points of all the candidate grabbing targets. According to the scheme, the grabbing points of the grabbing targets can be accurately recognized, the grabbing feasible targets are determined in combination with collision detection, follow-up grabbing is executed based on the sorting of the grabbing feasible targets, and the system operation efficiency is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method, apparatus, and electronic device for capturing images based on image segmentation results. Background Technology

[0002] With the development of industrial automation and intelligent manufacturing, vision-based automated sorting systems are widely used in the large-scale assembly, quality inspection, and classification of standard parts (such as screws and nuts). A typical grasping process includes steps such as image acquisition, target recognition, image segmentation, pose estimation, and grasping path planning.

[0003] In practical deployments, deep learning-based pose estimation algorithms can accurately identify the six-DOF pose of targets, especially for simple, standard-shaped objects like screws, achieving relatively stable pose inference. However, these algorithms suffer from several problems during inference, such as long runtime and high computational cost. This is particularly problematic when multiple targets are present in each image, leading to significant processing delays that hinder practical application. Furthermore, poor or failed pose estimation can easily cause grasping failures or even device malfunctions.

[0004] Therefore, pre-screening the screw targets obtained from image segmentation before pose estimation is a key step to improve system efficiency and stability.

[0005] In existing technologies, pre-screening of screw targets primarily involves obtaining the screw's center direction by fitting a circle to the target, then using the least squares method to fit and sum the centerline, and further solving for parameters such as the screw's gripping point and angle. This approach does not further optimize the extraction of the principal axis direction by incorporating actual density projection, and lacks automatic inference of target orientation and filtering of gripping strategies. Alternatively, some methods use image segmentation to obtain the target point cloud and then directly calculate the OBB envelope and 6D pose of the target workpiece using the Ransac and PCA algorithms. However, this method has low computational efficiency in multi-target scenarios and lacks collision region detection, easily leading to gripping failures. Summary of the Invention

[0006] The purpose of this invention is to provide a grasping method, apparatus, and electronic device based on image segmentation results, so as to accurately grasp the grasping points of the target and improve the system operating efficiency.

[0007] In a first aspect, the present invention provides a method for capturing images based on image segmentation results, the method comprising: A mask image containing the carrier and multiple grasping targets is obtained, and the foreground region of the mask image is extracted to obtain a foreground mask image containing the multiple grasping targets; The foreground mask image is segmented to obtain sub-images containing each grasping target; For each of the sub-images, define the smallest bounding rectangle of the target to be captured in the sub-image; The main axis is determined based on the minimum circumscribed rotation rectangle, and the target to be grasped is projected onto the pixel points based on the main axis. The grasping point is determined based on the density information of the projected points. Based on the grasping points of each grasping target, non-collision candidate grasping targets are selected; The selected candidate grasping targets are sorted to obtain a grasping order, and the robotic arm is controlled to perform grasping according to the grasping order and based on the grasping points of each candidate grasping target.

[0008] In an optional implementation, the step of projecting pixels onto the grasping target based on the main axis and determining the grasping point based on the density information of the projected points includes: Each pixel on the target to be grasped is projected onto the main axis to obtain a projection point; The projection points on the main axis are divided into multiple groups, and the number of projection points in each group is counted to construct a projection histogram. The density information of the projection points is determined based on the projection histogram, and the grasping point of the grasping target is determined based on the density information.

[0009] In an optional implementation, the step of determining the density information of the projection points based on the projection histogram, and determining the grasping points of the grasping target based on the density information, includes: The projection histogram is filtered to obtain the density information of the projection points after filtering. The density information includes the group with the most projection points and its neighboring groups. The offset value is calculated based on the number of projection points in the group containing the most projection points and the groups next to it. The grab point for the grabbing target is calculated based on the center point and offset value of the minimum circumscribed rotation rectangle and the unit vector of the principal axis.

[0010] In an optional implementation, the unit vector of the principal axis is determined in the following way: The projection histogram is divided into two parts, and the number of projection points in each part is counted. The direction of the main axis is determined to be from the part with a larger number of projection points to the part with a smaller number of projection points; The unit vector of the principal axis is determined based on the principal axis direction.

[0011] In an optional implementation, the step of filtering out collision-free candidate grasping targets based on the grasping points of each grasping target includes: Extract the background region of the mask image to obtain a background mask image containing the carrier, and determine the invalid grabbing region in the background mask image; Based on the grasping point of the grasping target and the center point of its smallest circumscribed rotation rectangle, a collision detection area is constructed; If the target being grasped overlaps with the invalid grasping area or the collision detection area of ​​other grasping targets, it is determined that the target being grasped has collision potential; otherwise, the target being grasped is determined to be a candidate target for grasping without collision potential.

[0012] In an optional implementation, the step of determining the invalid grabbing region in the background mask image includes: Construct the minimum bounding rectangle of the carrier in the background mask image; The minimum bounding rectangle is shrunk based on the set shrinkage factor to obtain the shrunk bounding rectangle; Extract the region between the minimum bounding rectangle and the bounding rectangle, and designate it as the invalid crawling region.

[0013] In an optional implementation, the step of constructing a collision detection region based on the grasping point of the grasping target and the center point of its smallest circumscribed rotation rectangle includes: A rectangular collision detection area is constructed based on the center point of the minimum circumscribed rotation rectangle of the target to be grasped and the set magnification factor; Based on the grasping point, the set magnification factor, and the length of the short side of the minimum circumscribed rotating rectangle, a circular collision detection area is constructed. The overlapping area between the rectangular collision detection area and the circular collision detection area is obtained as the collision detection area.

[0014] In an optional implementation, the step of sorting the selected candidate crawling targets to obtain the crawling order includes: Calculate the distance between the grab point of each selected candidate grab target and the center point of the mask image, and obtain the confidence level of the classification category of each candidate grab target; The reciprocal of the distance and the confidence level are summed according to the set weights to obtain the capture score; The crawling order is obtained by sorting multiple candidate crawling targets according to their crawling scores.

[0015] Secondly, the present invention provides a grasping device based on image segmentation results, the device comprising: The acquisition module is used to acquire a mask image containing a carrier and multiple grasping targets, and extract the foreground region of the mask image to obtain a foreground mask image containing the multiple grasping targets; The segmentation processing module is used to segment the foreground mask image to obtain sub-images containing each grasping target; The delineation module is used to delineate the minimum bounding rotation rectangle of the target to be captured in each of the sub-images. The determination module is used to determine the main axis based on the minimum circumscribed rotation rectangle, project the pixels of the grasping target based on the main axis, and determine the grasping point based on the density information of the projected points; The filtering module is used to filter out non-collision candidate targets based on the grasping points of each of the grasping targets; The control module is used to sort the selected candidate grasping targets to obtain a grasping order, and control the robotic arm to perform grasping according to the grasping order and based on the grasping points of each candidate grasping target.

[0016] Thirdly, the present invention provides an electronic device including one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine-executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-executable instructions to perform the method described in any of the foregoing embodiments.

[0017] This invention provides a grasping method, apparatus, and electronic device based on image segmentation results. The method involves acquiring a mask image containing a carrier and multiple grasping targets, extracting the foreground region of the mask image to obtain a foreground mask image containing multiple grasping targets, segmenting the foreground mask image to obtain sub-images containing each grasping target, dividing the sub-images into minimum bounding rectangles of rotation for each grasping target, determining the principal axis based on the minimum bounding rectangle, projecting pixel points onto the grasping targets based on the principal axis, and determining grasping points based on the density information of the projected points. Based on the grasping points of each grasping target, non-collision-prone candidate grasping targets are selected, and the selected candidate grasping targets are sorted to obtain a grasping order. The robotic arm is then controlled to perform grasping according to the grasping order and based on the grasping points of each candidate grasping target. This solution can accurately identify the grasping points of the grasping targets and, combined with collision detection, determine feasible grasping targets. Subsequent grasping is then performed based on the sorting of feasible grasping targets, significantly improving system operating efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating the image segmentation-based capture method provided in this embodiment of the invention; Figure 2 for Figure 1 A flowchart of the sub-steps included in S14; Figure 3 for Figure 2 A flowchart of the sub-steps included in S143; Figure 4 for Figure 1 A flowchart of the sub-steps included in S15; Figure 5 for Figure 4 A flowchart of the sub-steps included in S152; Figure 6 This is a schematic diagram of the target to be grasped in an embodiment of the present invention; Figure 7 for Figure 1 A flowchart of the sub-steps included in S16; Figure 8 A functional block diagram of the image segmentation-based grasping device provided in an embodiment of the present invention; Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0021] Please see Figure 1 The above is a flowchart of a grasping method based on image segmentation results provided in an embodiment of the present invention. The grasping method based on image segmentation results can be executed by a grasping device based on image segmentation results. The grasping device based on image segmentation results can be implemented by software and / or hardware and can be configured in an electronic device, such as a computer device, a control device communicating with a robotic arm, or a control device inside the robotic arm. The detailed steps of the grasping method based on image segmentation results are described below.

[0022] S11, acquire a mask image containing the carrier and multiple grasping targets, and extract the foreground region of the mask image to obtain a foreground mask image containing the multiple grasping targets.

[0023] S12, the foreground mask image is segmented to obtain sub-images containing each grasping target.

[0024] S13, for each of the sub-images, define the minimum circumscribed rotation rectangle of the target to be captured in the sub-image.

[0025] S14, determine the main axis based on the minimum circumscribed rotation rectangle, and project pixel points onto the grasping target based on the main axis, and determine the grasping point based on the density information of the projected points.

[0026] S15, based on the grasping points of each grasping target, select non-collision candidate grasping targets.

[0027] S16, sort the selected candidate grasping targets to obtain a grasping order, and control the robotic arm to perform grasping according to the grasping order and based on the grasping points of each candidate grasping target.

[0028] In this embodiment, the carrier can be a pallet, and the grasping target can be a screw or nut. It should be noted that in other grasping scenarios, the carrier and the grasping target can be other objects, and this embodiment does not limit them.

[0029] Before controlling the robotic arm to grasp, images can be captured using a camera to obtain images containing the load and multiple grasping targets. The captured images are then masked to obtain corresponding mask images. The pixels in the obtained mask image can be represented as a binary mask. .

[0030] Foreground regions are extracted from the mask image to obtain a foreground mask image containing multiple grasping targets, which can be represented as follows:

[0031] The obtained foreground mask image contains multiple grasping targets. Therefore, the foreground mask image can be segmented to obtain sub-images containing only a single grasping target.

[0032] For the grasping target in each sub-image, a minimum bounding rectangle containing the grasping target is defined. In this embodiment, the `minAreaRect` method of OpenCV can be used to define the minimum bounding rectangle, which can be represented as follows:

[0033] in, The center point of the smallest circumscribed rectangle of rotation. The two sides are the long side and the short side, respectively. Let be the rotation angle relative to the horizontal axis. Taking the longer side of the smallest circumscribed rectangle as the principal axis, and denoting the unit vector of the longer side's direction as:

[0034] By projecting each pixel in the target onto the main axis, the grab point of the target can be determined based on the density of the projected points on the main axis.

[0035] Since collisions between the grasping target and other grasping targets or objects should be avoided during the grasping process, collision detection is also required. In this embodiment, the grasping point of each grasping target is used to detect whether a collision is possible. If a collision is possible, the grasping target will not be used as the grasping object in this instance. In this way, collision-free grasping targets, i.e., those that will not collide, can be selected as candidate grasping targets.

[0036] To improve the convenience and efficiency of the robotic arm's grasping operation, this embodiment further sorts the selected candidate grasping targets to obtain a grasping order. This allows the robotic arm to grasp each candidate target according to the grasping order, specifically based on the grasping points of each candidate target during the grasping process.

[0037] Please see Figure 2 In this embodiment, the steps of projecting pixels onto the target based on the main axis and determining the grasping point based on the density information of the projected points can be implemented in the following way: S141, Project each pixel on the grasping target onto the main axis to obtain a projection point; S142, the projection points on the main axis are divided into multiple groups, and the number of projection points in each group is counted to construct a projection histogram; S143, determine the density information of the projection points based on the projection histogram, and determine the grasping point of the grasping target based on the density information.

[0038] In this embodiment, each pixel on the target is projected onto the main axis to obtain projection points. The projection coordinates of each projection point on the main axis can be represented as follows:

[0039] In this way, the minimum and maximum projected coordinate values ​​can be determined. Then, within the interval of the minimum and maximum projected coordinate values ​​[ The projection points are divided into multiple bins (e.g., N groups) with equal intervals within each bin. The number of projection points in each group is counted to construct a projection histogram. , can be represented as follows:

[0040] The constructed projection histogram can represent the density information of the projection points, including their distribution along the principal axis. Based on this density information, the grasping points for the grasping target are determined. For details, please refer to [link to relevant documentation]. Figure 3 The density information of the projected points can be determined based on the projection histogram, and the grasping points of the grasping target can be determined based on the density information: S1431, the projection histogram is filtered to obtain the density information of the projection points after filtering. The density information includes the group with the most projection points and its neighboring groups. S1432, the offset value is calculated based on the number of projection points in the group containing the most projection points and the groups next to it; S1433, calculate the grab point for the grab target based on the center point and offset value of the minimum circumscribed rotation rectangle and the unit vector of the principal axis.

[0041] In this embodiment, the projection histogram can be filtered using a one-dimensional Gaussian filter to remove noise. The filtered projection histogram... It can be represented as follows:

[0042] in, These are the filtering parameters involved in one-dimensional Gaussian filtering.

[0043] To obtain the density information of the projected points after filtering, we mainly obtain the index of the group containing the most projected points after filtering. Then, based on the number of points in the group containing the most projected points and the number of projected points in its neighboring groups, we calculate an offset value, as follows:

[0044] in, Indicates the index of the group containing the most projected points. This represents the number of projected points in the group containing the most projected points. This indicates the number of projection points within a neighboring group. This represents the calculated offset value.

[0045] Based on this, the grasping point of the grasping target is calculated according to the center point of the minimum circumscribed rotation rectangle, the offset value, and the unit vector of the principal axis. , can be represented as follows:

[0046] To facilitate the robotic arm in determining the direction of grasping, the unit vector of the main axis needs to include the main axis direction information. Based on this, in this embodiment, the unit vector of the main axis can be determined in the following way: The projection histogram is divided into two parts, and the number of projection points in each part is counted; the principal axis direction is determined to be from the part with a larger number of projection points to the part with a smaller number of projection points; the unit vector of the principal axis is determined based on the principal axis direction.

[0047] In the screw gripping scenario, the screw head has more pixels than other positions, and correspondingly, there are more projection points after projection. Therefore, by dividing the projection histogram into two parts and counting the number of projection points in each part, it can be determined that the part with a larger number of projection points corresponds to the screw head, while the part with a smaller number of projection points corresponds to the screw tail.

[0048] The direction of the spindle can be determined as a direction from the part with a larger number of projection points to the part with a smaller number of projection points, that is, for example, from the head of the screw to the tail of the screw.

[0049] Once the principal axis direction is determined, the unit vector of the principal axis can be determined. Then, following the calculation method described above, the grasping point of the grasping target can be calculated.

[0050] After determining the grab points for each target, it is also necessary to determine whether collisions are possible between the targets. Please refer to [link / reference]. Figure 4 In this embodiment, non-collision candidate targets can be selected based on the grasping points of each grasping target. Specifically, this can be achieved in the following way: S151, extract the background region of the mask image to obtain a background mask image containing the carrier, and determine the invalid grabbing region in the background mask image; S152, Based on the grasping point of the grasping target and the center point of its smallest circumscribed rotation rectangle, construct a collision detection area; S153, detect whether the grasping target has an overlapping area with the invalid grasping area and the collision detection area of ​​other grasping targets. If so, execute the following step S154; otherwise, execute the following step S155.

[0051] S154, It is determined that the target being grasped is collidable.

[0052] S155, determine the grasping target as a non-collision candidate grasping target.

[0053] In this embodiment, a similar method is used to extract the background region from the obtained mask image to obtain a background mask image containing the carrier, such as a tray. To allow for a gripping gap between the gripper and the tray boundary, the edge position on the tray is defined as an invalid gripping area. That is, if the target is located within the invalid gripping area, it will not be considered for gripping. In this embodiment, this situation is also included in the collision behavior of the gripping target; that is, if the target overlaps with the invalid gripping area, a collision is considered to have occurred.

[0054] In this embodiment, the invalid grabbing area in the background mask image is determined, which can be achieved in the following way: Construct the minimum bounding rectangle of the carrier in the background mask image; reduce the minimum bounding rectangle based on the set reduction factor to obtain the reduced bounding rectangle; extract the area between the minimum bounding rectangle and the bounding rectangle as the invalid grab area.

[0055] In this embodiment, the minimum bounding rectangle of the carrier is constructed in a similar manner to that described above, and the boundary of the minimum bounding rectangle can be T. A scaling factor is set. For example, the value can be 0.8 or any other value. Based on the shrinkage factor, the minimum bounding rectangle is shrunk. The shrunk bounding rectangle can be... The region between the original minimum bounding rectangle and the reduced bounding rectangle is considered the invalid crawling region, which can be represented as: .

[0056] In addition to detecting whether the target to be grabbed will collide with an invalid grabbing area, it is also necessary to detect whether the target to be grabbed will collide with other targets to be grabbed.

[0057] In this embodiment, a collision detection region can be constructed based on the grasping points of each grasping target and the center point of its smallest circumscribed rotation rectangle. For details, please refer to [link to relevant documentation]. Figure 5 This can be achieved in the following ways: S1521, Construct a rectangular collision detection area based on the center point of the smallest circumscribed rotation rectangle of the grasping target and the set magnification factor; S1522, Based on the grasping point, the set magnification factor, and the length of the short side of the minimum circumscribed rotating rectangle, a circular collision detection area is constructed; S1523, Obtain the overlapping area of ​​the rectangular collision detection area and the circular collision detection area as the collision detection area.

[0058] In this embodiment, the center point c of the smallest circumscribed rotation rectangle of the target to be grasped is used as the reference, based on the magnification factor. The construction magnification ratio is rectangular collision detection area , means as follows:

[0059] By grab point Using the center point as the radius, the radius is calculated based on the magnification factor and the length of the shorter side of the smallest circumscribed rectangle. .

[0060] Construct a circular collision detection area using the set center point and radius. , can be represented as follows:

[0061] Then, the intersection of the rectangular collision detection region and the circular collision detection region, that is, the overlapping area of ​​the two, is taken as the final collision detection region. , can be represented as follows:

[0062] After determining the collision detection region for each grasping target, a pixel-level AND operation is performed between the grasping target, the invalid grasping region, and the collision detection regions of other grasping targets (which can be rectangular to ensure collision avoidance). This checks for overlapping regions; if overlapping regions exist, a collision is possible; otherwise, collision-free conditions are determined. The selected candidate grasping targets can be as follows: Figure 6 As shown in the image.

[0063] After filtering out all non-collision candidate targets, sort the candidate targets. Please refer to [link / reference]. Figure 7 In this embodiment, the step of sorting the selected candidate crawling targets to obtain the crawling order can be implemented in the following way: S161, calculate the distance between the grab point of each selected candidate grab target and the center point of the mask image, and obtain the confidence level of the classification category of each candidate grab target; S162, the reciprocal of the distance and the confidence level are summed according to the set weights to obtain the capture score; S163, sort the multiple candidate targets according to their capture scores to obtain the capture order.

[0064] In this embodiment, the center point C of a mask image containing the carrier and multiple grasping targets can be located, and the Euclidean distance between the grasping point of each candidate grasping target and the center point can be calculated.

[0065] Furthermore, when obtaining the mask image, a pre-trained classification model can be used to identify the classification category of each grasping target and its corresponding confidence level. In this way, the confidence level of the classification category of each candidate grasping target can be obtained. .

[0066] The distance between the capture point of each candidate target and the center point of the mask image can be normalized. Since candidate targets closer to the center point should be given higher priority, and candidate targets with higher confidence scores should also be given higher priority, the reciprocal of the distance can be taken here. The reciprocal of the distance and the confidence score are then added together according to the set weights to obtain the capture score. The calculation method is as follows:

[0067] Here, w1 and w2 represent weights, such as 0.8 and 0.2 respectively.

[0068] In this way, multiple candidate grasping targets can be sorted according to their grasping scores from highest to lowest to obtain a grasping order. The robotic arm can then be controlled to perform the grasping according to this grasping order and based on the grasping points of each candidate grasping target. In addition, information such as the mask index, main axis direction, rectangular collision detection area, and circular collision detection area of ​​the candidate grasping targets can be provided to accurately calculate the 6D pose of the robotic arm's pose estimation model, and then execute the grasping action.

[0069] The image segmentation-based grasping method provided in this embodiment uses the minimum bounding rectangle to determine the initial direction of the target's principal axis, performs pixel-level projection analysis based on the principal axis direction, constructs a projection histogram, and uses Gaussian filtering to smooth the projection density to find the optimal grasping position. Furthermore, it employs a grasping region construction and collision detection strategy to dynamically construct enlarged rectangular and circular collision detection regions based on the principal axis direction. Boolean operations are then performed on these regions in conjunction with other target and obstacle masks to determine the presence of collisions.

[0070] This solution improves the overall efficiency of the grasping system. Instead of performing complex 6D pose estimation algorithms on every target in the image, it significantly reduces the amount of computation and inference time by efficiently preprocessing and filtering the segmentation mask, and only retains targets that are feasible for grasping in the subsequent processing.

[0071] Furthermore, the accuracy of spindle extraction and gripping point selection is enhanced. Compared with the traditional method of estimating the gripping direction through the minimum bounding rectangle, this invention introduces a mask projection density analysis mechanism. Combined with Gaussian filtering and projection histogram analysis, it can extract the screw spindle direction and gripping point more accurately and stably.

[0072] Furthermore, a controllable direction determination and area collision detection mechanism has been implemented. This invention can automatically determine the target's grasping direction and dynamically generate a grasping area model that is registered with it based on the main axis direction. It integrates collision detection of the environment and other objects to achieve a more reasonable grasping point feasibility determination.

[0073] Based on the same inventive concept, please refer to Figure 8 This invention also provides a functional module diagram of a grasping device based on image segmentation results. This embodiment can divide the grasping device based on image segmentation results into functional modules according to the above method embodiments. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment is illustrative and only represents a logical functional division; other division methods may be used in actual implementation.

[0074] For example, when dividing functional modules according to their respective functions, Figure 8 The image segmentation-based grasping device shown is only a schematic diagram. This device may include an acquisition module, a segmentation processing module, a delineation module, a determination module, a filtering module, and a control module. The functions of each module of this image segmentation-based grasping device will be described in detail below.

[0075] The acquisition module is used to acquire a mask image containing a carrier and multiple grasping targets, and extract the foreground region of the mask image to obtain a foreground mask image containing the multiple grasping targets; The segmentation processing module is used to segment the foreground mask image to obtain sub-images containing each grasping target; The delineation module is used to delineate the minimum bounding rotation rectangle of the target to be captured in each of the sub-images. The determination module is used to determine the main axis based on the minimum circumscribed rotation rectangle, project the pixels of the grasping target based on the main axis, and determine the grasping point based on the density information of the projected points; The filtering module is used to filter out non-collision candidate targets based on the grasping points of each of the grasping targets; The control module is used to sort the selected candidate grasping targets to obtain a grasping order, and control the robotic arm to perform grasping according to the grasping order and based on the grasping points of each candidate grasping target.

[0076] In one possible implementation, the aforementioned determining module can be used to determine the grab point in the following way: Each pixel on the target to be grasped is projected onto the main axis to obtain a projection point; The projection points on the main axis are divided into multiple groups, and the number of projection points in each group is counted to construct a projection histogram. The density information of the projection points is determined based on the projection histogram, and the grasping point of the grasping target is determined based on the density information.

[0077] In one possible implementation, the aforementioned determining module can be used to determine the grasping point of the grasping target based on density information in the following manner: The projection histogram is filtered to obtain the density information of the projection points after filtering. The density information includes the group with the most projection points and its neighboring groups. The offset value is calculated based on the number of projection points in the group containing the most projection points and the groups next to it. The grab point for the grabbing target is calculated based on the center point and offset value of the minimum circumscribed rotation rectangle and the unit vector of the principal axis.

[0078] In one possible implementation, the aforementioned determining module is used to determine the unit vector of the principal axis in the following manner: The projection histogram is divided into two parts, and the number of projection points in each part is counted. The direction of the main axis is determined to be from the part with a larger number of projection points to the part with a smaller number of projection points; The unit vector of the principal axis is determined based on the principal axis direction.

[0079] In one possible implementation, the above filtering module is used to filter candidate crawling targets by: Extract the background region of the mask image to obtain a background mask image containing the carrier, and determine the invalid grabbing region in the background mask image; Based on the grasping point of the grasping target and the center point of its smallest circumscribed rotation rectangle, a collision detection area is constructed; If the target being grasped overlaps with the invalid grasping area or the collision detection area of ​​other grasping targets, it is determined that the target being grasped has collision potential; otherwise, the target being grasped is determined to be a candidate target for grasping without collision potential.

[0080] In one possible implementation, the filtering module described above is used to determine invalid crawled regions in the following way: Construct the minimum bounding rectangle of the carrier in the background mask image; The minimum bounding rectangle is shrunk based on the set shrinkage factor to obtain the shrunk bounding rectangle; Extract the region between the minimum bounding rectangle and the bounding rectangle, and designate it as the invalid crawling region.

[0081] In one possible implementation, the above filtering module is used to construct the collision detection region in the following way: A rectangular collision detection area is constructed based on the center point of the minimum circumscribed rotation rectangle of the target to be grasped and the set magnification factor; Based on the grasping point, the set magnification factor, and the length of the short side of the minimum circumscribed rotating rectangle, a circular collision detection area is constructed. The overlapping area between the rectangular collision detection area and the circular collision detection area is obtained as the collision detection area.

[0082] In one possible implementation, the control module described above is used to obtain the grabbing order in the following way: Calculate the distance between the grab point of each selected candidate grab target and the center point of the mask image, and obtain the confidence level of the classification category of each candidate grab target; The distance and confidence level are summed according to the set weights to obtain the capture score; The crawling order is obtained by sorting multiple candidate crawling targets according to their crawling scores.

[0083] The image segmentation-based crawling device provided in this embodiment can be used to execute the image segmentation-based crawling method under any of the above embodiments. For details not covered in this embodiment, please refer to the corresponding descriptions in the above embodiments. This embodiment will not elaborate further here.

[0084] Please see Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device can be a computer device, a control device, or the like. The electronic device includes a memory, a processor, and a communication module. The memory, processor, and communication module are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0085] The memory is used to store computer programs or data. Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0086] The processor is used to read / write data or programs stored in the memory and execute the image segmentation-based capture method provided in any embodiment of the present invention.

[0087] The communication module is used to establish communication connections between electronic devices and other communication terminals via a network, and to send and receive data via the network.

[0088] It should be understood that, Figure 9 The structure shown is only a schematic diagram of an electronic device; the electronic device may also include components that are larger than those shown. Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown.

[0089] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions, which, when executed, implement the image segmentation-based capture method provided in the above embodiments.

[0090] Specifically, the computer-readable storage medium can be a general-purpose storage medium, such as a removable disk or hard disk. When the computer program on the computer-readable storage medium is run, it can execute the above-described image segmentation-based capture method. The processes involved in the execution of the executable instructions on the computer-readable storage medium can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0091] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0092] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0093] Furthermore, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0094] It should be noted that if the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0096] The above are merely embodiments of the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for capturing images based on image segmentation results, characterized in that, The method includes: A mask image containing the carrier and multiple grasping targets is obtained, and the foreground region of the mask image is extracted to obtain a foreground mask image containing the multiple grasping targets; The foreground mask image is segmented to obtain sub-images containing each grasping target; For each of the sub-images, define the smallest bounding rectangle of the target to be captured in the sub-image; The main axis is determined based on the minimum circumscribed rotation rectangle, and the target to be grasped is projected onto the pixel points based on the main axis. The grasping point is determined based on the density information of the projected points. Based on the grasping points of each grasping target, non-collision candidate grasping targets are selected; The selected candidate grasping targets are sorted to obtain a grasping order, and the robotic arm is controlled to perform grasping according to the grasping order and based on the grasping points of each candidate grasping target.

2. The image segmentation-based capture method according to claim 1, characterized in that, The step of projecting pixels onto the grasping target based on the main axis and determining the grasping point based on the density information of the projected points includes: Each pixel on the target to be grasped is projected onto the main axis to obtain a projection point; The projection points on the main axis are divided into multiple groups, and the number of projection points in each group is counted to construct a projection histogram. The density information of the projection points is determined based on the projection histogram, and the grasping point of the grasping target is determined based on the density information.

3. The image segmentation-based capture method according to claim 2, characterized in that, The step of determining the density information of the projection points based on the projection histogram, and determining the grasping points of the grasping target based on the density information, includes: The projection histogram is filtered to obtain the density information of the projection points after filtering. The density information includes the group with the most projection points and its neighboring groups. The offset value is calculated based on the number of projection points in the group containing the most projection points and the groups next to it. The grab point for the grabbing target is calculated based on the center point and offset value of the minimum circumscribed rotation rectangle and the unit vector of the principal axis.

4. The image segmentation-based capture method according to claim 3, characterized in that, The unit vector of the principal axis is determined in the following way: The projection histogram is divided into two parts, and the number of projection points in each part is counted. The direction of the main axis is determined to be from the part with a larger number of projection points to the part with a smaller number of projection points; The unit vector of the principal axis is determined based on the principal axis direction.

5. The image segmentation-based capture method according to claim 1, characterized in that, The step of filtering out collision-free candidate targets based on the grasping points of each of the grasping targets includes: Extract the background region of the mask image to obtain a background mask image containing the carrier, and determine the invalid grabbing region in the background mask image; Based on the grasping point of the grasping target and the center point of its smallest circumscribed rotation rectangle, a collision detection area is constructed; If the target being grasped overlaps with the invalid grasping area or the collision detection area of ​​other grasping targets, it is determined that the target being grasped has collision potential; otherwise, the target being grasped is determined to be a candidate target for grasping without collision potential.

6. The image segmentation-based capture method according to claim 5, characterized in that, The step of determining the invalid grabbing region in the background mask image includes: Construct the minimum bounding rectangle of the carrier in the background mask image; The minimum bounding rectangle is shrunk based on the set shrinkage factor to obtain the shrunk bounding rectangle; Extract the region between the minimum bounding rectangle and the bounding rectangle, and designate it as the invalid crawling region.

7. The image segmentation-based capture method according to claim 5, characterized in that, The step of constructing a collision detection region based on the grasping point of the grasping target and the center point of its smallest circumscribed rotation rectangle includes: A rectangular collision detection area is constructed based on the center point of the minimum circumscribed rotation rectangle of the target to be grasped and the set magnification factor; Based on the grasping point, the set magnification factor, and the length of the short side of the minimum circumscribed rotating rectangle, a circular collision detection area is constructed. The overlapping area between the rectangular collision detection area and the circular collision detection area is obtained as the collision detection area.

8. The image segmentation-based capture method according to claim 1, characterized in that, The step of sorting the selected candidate crawling targets to obtain the crawling order includes: Calculate the distance between the grab point of each selected candidate grab target and the center point of the mask image, and obtain the confidence level of the classification category of each candidate grab target; The reciprocal of the distance and the confidence level are summed according to the set weights to obtain the capture score; The crawling order is obtained by sorting multiple candidate crawling targets according to their crawling scores.

9. A grasping device based on image segmentation results, characterized in that, The device includes: The acquisition module is used to acquire a mask image containing a carrier and multiple grasping targets, and extract the foreground region of the mask image to obtain a foreground mask image containing the multiple grasping targets; The segmentation processing module is used to segment the foreground mask image to obtain sub-images containing each grasping target; The delineation module is used to delineate the minimum bounding rotation rectangle of the target to be captured in each of the sub-images. The determination module is used to determine the main axis based on the minimum circumscribed rotation rectangle, project the pixels of the grasping target based on the main axis, and determine the grasping point based on the density information of the projected points; The filtering module is used to filter out non-collision candidate targets based on the grasping points of each of the grasping targets; The control module is used to sort the selected candidate grasping targets to obtain a grasping order, and control the robotic arm to perform grasping according to the grasping order and based on the grasping points of each candidate grasping target.

10. An electronic device, characterized in that, The device includes one or more storage media and one or more processors communicating with the storage media. The one or more storage media store machine-executable instructions that are executable by the processor. When the electronic device is running, the processor executes the machine-executable instructions to perform the method according to any one of claims 1-8.

Images