Upper part extraction method and device, computer equipment and storage medium

Abstract

The present application relates to the technical field of computer vision, and provides a kind of upper part extraction method, device, computer equipment and storage medium, the method comprises: obtaining the point cloud of stacked piece, the point cloud is evenly divided into non-overlapping point cloud module;Each point cloud module is merged into plane based on mean square error, and initial segmentation plane is obtained;According to the distance between the initial segmentation plane and the adjacent point cloud of plane boundary, the initial segmentation plane is finely segmented, and finely segmented plane is obtained;Each finely segmented plane is compared and screened upper layer plane two by two, and the uppermost part plane is obtained by splicing each upper layer plane;According to the uppermost part plane, the uppermost part in the stacked piece is extracted.The method can improve efficiency and accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method, apparatus, computer device and storage medium for extracting upper-level parts. Background Technology

[0002] Automated storage and retrieval systems (AS / RS) play a crucial role in sorting, assembly, and processing tasks in smart factories. AS / RS equipment enables efficient high-level warehouse operations, automated storage and retrieval, and simplified operation. In heavy industry settings, factories store steel plate workpieces for welding and assembly in AS / RS. These workpieces are often stacked on crates. Once a production demand is received, the AS / RS moves the corresponding crate to the picking position. Computer vision technology guides the gantry crane to pick up the specific workpiece required for the order and place it on a conveyor belt. The conveyor belt then transports the workpiece to the other end, where computer vision again guides the gantry crane to pick it up and place it on the corresponding pallet. AGVs (Automated Guided Vehicles) then transport the workpiece to the designated workstation.

[0003] Due to the large weight, variety, and similarity of incoming materials, problems such as low work efficiency, mechanical collisions, and incorrect material handling can easily occur, significantly impacting the overall automation level of the factory. Therefore, automatic identification and handling of workpieces in vision-based automated warehouses (AS / RS) have become a key research focus in warehouse automation. However, due to significant variations in ambient lighting, the current use of ordinary 2D imaging for automated material handling is not very effective. Furthermore, the non-standard and often chaotic stacking of workpieces in the material frames can cause interference between upper and lower layers in the imaging, leading to identification errors. In addition, workpiece bending deformation and the influence of natural light can also cause image gaps, further reducing the accuracy of automatic workpiece identification and handling in AS / RS. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, and storage medium for extracting upper-layer parts of stacked components that can improve efficiency and accuracy in addressing the aforementioned technical problems.

[0005] This invention provides a method for extracting the upper layer component of a stacked assembly, comprising: Obtain the point cloud of the stacked components, and divide the point cloud evenly into non-overlapping point cloud modules; Based on the mean square error, the point cloud modules are merged into a plane to obtain the initial segmentation plane; The initial segmentation plane is finely segmented based on the distance between the initial segmentation plane and the point cloud adjacent to the plane boundary to obtain a finely segmented plane; Each of the finely segmented planes is compared and selected pairwise to form an upper-level plane, and the upper-level planes are then spliced ​​together to obtain the uppermost part plane. The topmost part in the stack is extracted based on the plane of the topmost part.

[0006] In one embodiment, after obtaining the point cloud of the stacked components and uniformly dividing the point cloud into non-overlapping point cloud modules, the method further includes: Point cloud modules with missing data or discontinuous depth are filtered out based on their depth values, and point cloud modules that do not meet the retention criteria are filtered out based on their mean square error and normal vector.

[0007] In one embodiment, filtering out point cloud modules that do not meet the retention criteria based on the mean square error of the point cloud modules includes: Convert each point cloud in the point cloud module into a point cloud matrix; Calculate the covariance matrix based on the point cloud matrix, obtain the eigenvalues ​​and eigenvectors of the covariance matrix, and fit the plane equation of the point cloud in the point cloud module based on the eigenvalues ​​and eigenvectors. The mean square error of each point cloud in the point cloud module is calculated based on the plane equation. Point cloud modules whose mean square error is less than the first error threshold are retained.

[0008] In one embodiment, filtering out point cloud modules that do not meet the retention criteria based on the normal vector of the point cloud module includes: Determine the normal vector of each point cloud in the point cloud module; When the point cloud module is determined to include the boundary between the upper and lower planes based on the difference between the normal vectors, the point cloud module is filtered out.

[0009] In one embodiment, merging the point cloud modules into a plane based on mean square error to obtain an initial segmentation plane includes: Among the unmerged point cloud modules, the point cloud module with the smallest mean square error is identified as the initial module; From the adjacent point cloud modules of the initial module, select point cloud modules whose mean square error after merging with the initial module is less than the second error threshold, merge them to obtain a new initial module, and then continuously select point cloud modules from the adjacent point cloud modules of the new initial module to continue merging until they cannot be merged, thus obtaining the initial segmentation plane. Return to the step of determining the point cloud module with the smallest mean square error among the unmerged point cloud modules as the initial module, until all the point cloud modules have been merged.

[0010] In one embodiment, the step of finely segmenting the initial segmentation plane based on the distance between the initial segmentation plane and the adjacent point clouds of the plane boundary to obtain a finely segmented plane includes: Determine the point clouds adjacent to each plane boundary of the initial segmentation plane, and calculate the distances to the point clouds adjacent to each plane boundary according to the plane equations of each point cloud in the initial segmentation plane; When the distance is less than the first distance threshold, the plane adjacent boundary point cloud corresponding to the distance does not belong to other planes, and the plane mean square error after the plane adjacent point cloud is included in the plane is less than the third error threshold, the plane adjacent point cloud is included in the initial segmentation plane. When the distance is less than the first distance threshold, the adjacent point cloud of the plane boundary corresponding to the distance belongs to another plane, and the mean square error of the merged plane is less than the third error threshold, the plane to which the adjacent point cloud of the plane boundary belongs is merged with the initial segmentation plane to obtain a fine segmentation plane.

[0011] In one embodiment, the step of comparing and selecting upper-layer planes pairwise from each of the finely segmented planes, and then splicing the upper-layer planes to obtain the uppermost part plane, includes: The horizontal plane and the inclined plane are determined based on the angle between each of the finely divided planes and the calibration plate plane; Compare each horizontal plane in pairs. When the distance between two horizontal planes is less than the second distance threshold and the angle difference between the two horizontal planes is less than the angle threshold, or the area of ​​the overlapping region of the two horizontal planes is greater than the first area threshold, retain the plane with the higher position from the two horizontal planes to obtain the upper plane. Each inclined plane is compared pairwise. When the distance between two inclined planes and the area of ​​the overlapping region of the two inclined planes are greater than the second area threshold, the plane with the higher position is retained from the two inclined planes to obtain the upper plane. Each horizontal plane and each inclined plane are compared pairwise. When the distance between the horizontal plane and the inclined plane is greater than or equal to a second distance threshold and the area of ​​the overlapping region between the horizontal plane and the inclined plane is greater than a third area threshold, the plane with the higher position is retained from the horizontal plane and the inclined plane to obtain the upper plane. By splicing together the aforementioned upper-layer planes, the uppermost part plane is obtained.

[0012] An upper-layer component extraction device for a stacked assembly includes: A point cloud partitioning module is used to acquire the point cloud of the stacked components and uniformly divide the point cloud into non-overlapping point cloud modules. The initial segmentation module is used to merge the point cloud modules into a plane based on the mean square error to obtain the initial segmentation plane; The fine segmentation module is used to finely segment the initial segmentation plane based on the distance between the initial segmentation plane and the adjacent point clouds of the plane boundary, so as to obtain a fine segmentation plane; The plane extraction module is used to compare and filter the upper planes of each finely segmented plane in pairs, and to stitch the upper planes together to obtain the uppermost part plane. The parts extraction module extracts the topmost part from the stacked components based on the plane of the topmost part.

[0013] The present invention also provides a computer device, the computer device including a processor and a memory, the memory storing a computer program, the processor executing the computer program to implement the steps of the method for extracting upper part of stacked components as described in any of the preceding claims.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method for extracting upper part of a stack as described in any of the preceding claims.

[0015] The aforementioned method, apparatus, computer equipment, and storage medium for extracting the upper-layer parts of stacked components involve uniformly dividing a point cloud captured by a 3D camera into multiple small point cloud modules. Then, based on the mean square error, point cloud modules considered to belong to the same plane are integrated to obtain an initial segmentation plane. Further fine segmentation is performed based on the distance between the initial segmentation plane and adjacent point clouds at the plane boundary to obtain a more accurate fine segmentation plane. Finally, by comparing pairs of fine segmentation planes, the upper-layer planes are selected and stitched together to obtain the upper-layer part plane for extraction. This method achieves imaging of the uppermost part based on 3D captured point clouds, featuring a large field of view, fast response speed, and wide environmental adaptability, remaining unaffected even in dark environments. Simultaneously, through point cloud merging, initial segmentation, and fine segmentation steps, the requirements for part stacking are reduced. Even with non-standard part stacking or workpiece bending deformation, the plane of the uppermost part can be accurately extracted and imaged, thereby improving the accuracy of subsequent automatic grasping. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the process environment of a method for extracting upper-layer components from a stack in one embodiment.

[0017] Figure 2 This is a schematic diagram of the point cloud normal vectors in one embodiment.

[0018] Figure 3 This is a schematic diagram of the initial segmentation plane in one embodiment.

[0019] Figure 4 This is a detailed planar diagram of the stacked components in one embodiment.

[0020] Figure 5 This is a schematic diagram of the plane of the topmost component in one embodiment.

[0021] Figure 6 This is a structural block diagram of an upper part extraction device for a stacked component in one embodiment. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0023] In one embodiment, such as Figure 1 As shown, a method for extracting the upper layer of a stacked component is provided, including the following steps: Step S101: Obtain the point cloud of the stacked components and divide the point cloud evenly into non-overlapping point cloud modules.

[0024] Specifically, the preferred method is to acquire a depth map of the stacked components using a 3D camera, convert this depth map data into a point cloud, and then uniformly divide the point cloud into multiple non-overlapping modules to obtain point cloud modules. During the division process, it is preferable to divide the obtained point cloud data into multiple 10x10 modules of a certain size, for example, 10x10 pixels. Furthermore, after obtaining the point cloud modules, they can be filtered based on actual needs, retaining only the required modules.

[0025] In one embodiment, step S101 is followed by a step of filtering point cloud modules, including: filtering out point cloud modules with missing data and discontinuous depth based on the depth value of the point cloud, and filtering out point cloud modules that do not meet the retention conditions based on the mean square error and normal vector of the point cloud modules.

[0026] Specifically, due to environmental influences or technical limitations of the 3D camera itself, some point cloud data may be missing. Therefore, point cloud modules with missing data or discontinuous depth data can be filtered out based on their depth values. For example, if a point cloud module contains many points with empty or zero depth values, it is considered to have severely missing data and can be filtered out. Alternatively, if the maximum and minimum depth values ​​of each point cloud in a module indicate significant variations in depth values ​​or discontinuous depth data, the module can also be filtered out. Furthermore, point cloud modules that do not belong to the same plane can be filtered out based on their mean square error (MSE). Point cloud modules that form the boundary between two planes can also be filtered out based on their normal vectors. The MSE of the point cloud can be calculated by fitting a point cloud formula using principal component analysis.

[0027] In one embodiment, filtering out point cloud modules that do not meet the retention criteria based on the mean square error of the point cloud modules includes: converting each point cloud in the point cloud module into a point cloud matrix; calculating a covariance matrix based on the point cloud matrix, obtaining the eigenvalues ​​and eigenvectors of the covariance matrix, fitting a plane equation of the point clouds in the point cloud module based on the eigenvalues ​​and eigenvectors; calculating the mean square error of each point cloud in the point cloud module based on the plane equation; and retaining point cloud modules whose mean square error is less than a first error threshold.

[0028] Specifically, based on the principles of Principal Component Analysis (PCA), for a large number of discrete point clouds, we first find their data center, and then find a 3D line based on this center point. The variance of the point cloud data projected onto this line is maximized, meaning the projection of all data along this direction is most dispersed, implying that more information is preserved. This line becomes the first principal component of the point cloud. Similarly, after finding the first principal component, we use the same method to find two other 3D lines in the 3D point cloud data; these constitute the second and third components of the point cloud. The point cloud data exhibits the least variation along the third component 3D line.

[0029] According to the plane equation formula: [A,B,C] represents the normal vector of the plane, meaning that the data in the plane changes most slowly along the normal vector, which coincides with the concept of the third principal component mentioned above. Therefore, by simply obtaining the third principal component of the corresponding point cloud using PCA, the formula for the point cloud in this point cloud module can be fitted. The specific process of solving for each principal component in the point cloud in this embodiment is as follows: First, convert each point cloud in the point cloud module into a point cloud matrix. Assuming there are N 3D point clouds (x, y, z) in the point cloud module, write all the 3D points as a 3×N matrix, then we have the following point cloud matrix A: Next, the covariance matrix of the point cloud matrix is ​​calculated. The covariance matrix represents the similarity of the x, y, and z values ​​of each point in the point cloud as they change. The covariance matrix is ​​calculated as follows: Finally, the eigenvalues ​​and eigenvectors of the covariance matrix are obtained using the following formulas, where λ represents the eigenvalues ​​of the matrix and E represents the eigenvectors of the matrix. A Representing the covariance matrix: A typical 3x3 covariance matrix has three eigenvalues, and the eigenvector corresponding to the smallest eigenvalue is the normal vector of the plane. Therefore, by combining the values ​​of the point cloud center points, we can fit the plane equation of the point cloud in this module. After fitting the plane equation, the mean square error of the point cloud can be calculated using the MSE (mean squared error) formula. That is, the average error between the actual point cloud and the theoretical position of the point cloud calculated from the fitted plane equation can be calculated using the MSE formula. If the error of each point cloud in a point cloud module is large, it means that the point clouds in that module do not belong to the same plane, and the point cloud module can be discarded. Whether the error is large can be determined by comparing it with a set error threshold. When the mean square error is less than the set first error threshold, it means that the error is small, indicating that the point clouds in the module should all belong to the same plane, and therefore can be retained; otherwise, the error is large and can be filtered out. The first error threshold is a pre-set error threshold that can be set according to the actual situation.

[0030] In one embodiment, filtering out point cloud modules that do not meet the retention criteria based on the normal vectors of the point cloud modules includes: determining the normal vectors of each point cloud in the point cloud module; and filtering out point cloud modules when the difference between the normal vectors determines that the point cloud module includes the boundary between the upper and lower planes.

[0031] Specifically, due to inherent quality issues with 3D cameras or the thinness of components like steel plates, the difference between upper and lower planes may be minimal. In such cases, it's impossible to remove the point cloud module containing the boundary between the two planes based solely on depth values ​​and mean square error. Therefore, this embodiment introduces normal vectors for judgment, effectively identifying the transition region between two planes. The difference between the normal vector of the transition region and the normal vectors of the two planes in the upper and lower layers is generally significant. Thus, based on the difference between the normal vectors of each point cloud in the point cloud module, it can be determined whether the transition region between the two planes is included. Figure 2 As shown, a schematic diagram of a point cloud normal vector is provided. Figure 2 The intermediate region shown is the angled area (transition region) between the two plane layers. If the point cloud module is determined to include such a transition region through the normal vector, then that point cloud module should be discarded.

[0032] Step S102: Based on the mean square error, merge each point cloud module into a plane to obtain the initial segmentation plane.

[0033] Specifically, after the point cloud modules are divided, they are integrated to achieve the initial segmentation of the plane. During this initial segmentation, the point cloud module with the smallest mean square error is selected as the initial module and merged with its surrounding modules until no further expansion is possible. Using the same method, the module with the smallest mean square error from the unintegrated modules is selected as the initial module and merged with it until all point cloud modules are integrated.

[0034] In one embodiment, step S102 includes: determining the point cloud module with the smallest mean square error among the unmerged point cloud modules as the initial module; selecting point cloud modules from the adjacent point cloud modules of the initial module whose mean square error after merging with the initial module is less than a second error threshold, merging them to obtain a new initial module, and cyclically selecting point cloud modules from the adjacent point cloud modules of the new initial module to continue merging until merging is no longer possible, thereby obtaining an initial segmentation plane; returning to the step of determining the point cloud module with the smallest mean square error among the unmerged point cloud modules as the initial module, until all point cloud modules have been merged.

[0035] Specifically, first, we find the point cloud module with the smallest MSE (mean squared error) as the initial module. Then, we merge and judge an initial module with its neighboring point cloud modules. The number of neighboring point cloud modules can be selected based on actual needs, such as 4 or 8. In this embodiment, to improve speed while ensuring quality, it is preferred to merge and judge 4 neighboring point cloud modules simultaneously.

[0036] If the initial module meets the merging requirement with one of its adjacent point cloud modules (i.e., the MSE of the resulting plane after merging is less than a certain threshold, such as a preset second error threshold), then it is merged with that adjacent point cloud module to form a new larger module. This process is repeated continuously, expanding the number of point clouds in the initial module from surrounding adjacent point cloud modules, thereby merging point cloud modules belonging to the same plane into a larger module, continuously expanding the corresponding plane range until it can no longer be expanded. In other words, if the current initial module completes the first merging to obtain a new larger module, this new larger module is regarded as the new initial module, and modules are selected and merged from surrounding adjacent point cloud modules in the same way until no new modules can be merged, meaning the plane corresponding to the currently selected initial module can no longer be expanded. The second error threshold is the same as the first error threshold and is a preset error threshold; it may or may not be equal to the first error threshold.

[0037] Then, when it is determined that the plane corresponding to the current initial module has expanded to its limit, a new initial module is selected. The process returns to selecting the module with the smallest MSE from the currently unmerged point cloud modules as the initial module, and the same merging process is repeated until all point cloud modules have been integrated into a plane, resulting in the initial segmentation plane. For example, suppose there are a total of 5 point cloud modules A, B, C, D, and E. When initially selecting the initial module, since all point cloud modules have not been merged, module A with the smallest MSE is selected as the initial module. Then, by merging A and B, a larger module AB is obtained. Using module AB as the new initial module, module D is merged to obtain module ABD. If it has expanded to its limit, the merging of module A is stopped. At this point, the plane formed by modules ABD is the current initial segmentation plane. Next, a new initial module is selected to continue merging, that is, the module with the smallest MSE is selected from the unmerged modules C and E as the initial module, and the merging and expansion are carried out in the same way as module A, until all point cloud modules have been merged, resulting in all the initial segmentation planes. Figure 3 As shown, a schematic diagram of the initial segmentation plane is provided. Figure 3 Different grayscale values ​​in the region represent different initial segmentation planes.

[0038] Step S103: The initial segmentation plane is finely segmented based on the distance between the initial segmentation plane and the adjacent point clouds of the plane boundary to obtain the fine segmentation plane.

[0039] Specifically, after initial segmentation, jagged edges may exist at the plane boundaries, and data loss may occur at boundary points, resulting in the acquired plane portion failing to reflect the actual state of the plane. Furthermore, if a part is occluded, the retained point cloud due to over-segmentation is extremely limited and also fails to reflect the true state. Therefore, after initial plane segmentation, this embodiment further performs fine segmentation. Fine segmentation not only solves the above problems but also recovers discarded valid point clouds. For example, point clouds discarded based on normal vectors, including those from overly segmented areas, may contain point clouds belonging to the topmost part plane, which can be recovered through fine segmentation. In this embodiment, fine segmentation primarily uses region growing to merge point clouds around the initial segmented plane. That is, the distance between each initial segmented plane and the adjacent point clouds at the plane boundary is used to determine whether the point cloud needs to be included in the plane, thus obtaining the finely segmented plane.

[0040] In one embodiment, step S103 includes: determining the adjacent point clouds of each plane boundary of the initial segmentation plane; calculating the distance to the adjacent point clouds of each plane boundary according to the plane equation of each point cloud in the initial segmentation plane; when the distance is less than a first distance threshold, the adjacent point cloud of the plane boundary corresponding to the distance does not belong to other planes, and the mean square error of the plane after the adjacent point cloud of the plane boundary is included in the plane is less than a third error threshold, the adjacent point cloud of the plane boundary is included in the initial segmentation plane; when the distance is less than the first distance threshold, the adjacent point cloud of the plane boundary corresponding to the distance belongs to other planes, and the mean square error of the plane after the plane is merged is less than the third error threshold, the plane to which the adjacent point cloud of the plane boundary belongs is merged with the initial segmentation plane to obtain a fine segmentation plane.

[0041] Specifically, for a given initial segmentation plane, the four adjacent point clouds (up, down, left, and right) of each boundary point in that plane region are considered as adjacent point clouds of the plane boundary, and a planar association calculation is performed between the initial segmentation plane and the initial segmentation plane. The specific steps of the planar association calculation are as follows: The plane equation of the initial segmentation plane is fitted, and the distance between the initial segmentation plane and the adjacent point clouds of the plane boundary is calculated using the point-to-plane distance formula. If the distance between the adjacent point clouds of the plane boundary and the initial segmentation plane is less than a preset first distance threshold, and the adjacent point clouds of the plane boundary are not on any other plane, then the point is directly included in the initial segmentation plane. However, if the distance is less than the first distance threshold, but the adjacent point clouds of the plane boundary are on another plane (i.e., the adjacent point clouds of the plane boundary belong to a plane), then the initial segmentation plane and this belonging plane are connected, and the initial segmentation plane and the belonging plane are merged into a new larger plane. At this point, a planar association calculation can be performed again between the adjacent point clouds of the plane boundary and the merged new larger plane. If it is still determined that the point belongs to the merged new larger plane, then the point is determined to be added to the region and boundary point of the new larger plane. If not, the two planes remain separate planes, are not merged, and the point is not included in the initial segmentation plane, thus completing the fine segmentation and obtaining the fine segmentation plane. Additionally, it's important to note that each time the point cloud is integrated into a plane, the plane equation and the mean square error (MSE) of the new integrated plane need to be recalculated. If the MSE is less than a certain threshold, such as a preset third error threshold, then the point is included in the plane; otherwise, it is not integrated. Figure 4 As shown, a detailed planar diagram of a stacked component is provided. Figure 4 Figure (a) shows the initial dividing plane of the stacked components, and Figure (b) shows the finely divided plane after finely dividing Figure (a). The third error threshold is also a pre-set error threshold, which can be equal to or different from the first or second error threshold. The first distance threshold is used to determine the distance between the planes and can also be pre-set according to the actual situation.

[0042] Step S104: Compare and select the upper planes from each finely divided plane, and splice the upper planes to obtain the uppermost part plane.

[0043] Specifically, although the most accurate segmentation result has been obtained after fine segmentation, the reference... Figure 4 As can be seen, the curved surfaces of the parts and the planes corresponding to the upper and lower parts are still connected, superimposed, or close to each other. Therefore, it is necessary to further filter and extract the plane corresponding to the topmost part. In this embodiment, each finely segmented plane is compared pairwise, and the plane located on the upper layer is retained by selecting one of the two. All the finely segmented planes that are finally retained are the planes corresponding to the topmost part. Then, all the retained upper-layer planes are spliced ​​together to obtain the plane of the topmost part.

[0044] In one embodiment, step S104 includes: determining horizontal planes and inclined planes based on the angles between each finely segmented plane and the calibration plate plane; comparing each horizontal plane pairwise, and when the distance between two horizontal planes is less than a second distance threshold and the angle difference between the two horizontal planes is less than an angle threshold, or the overlapping area of ​​the two horizontal planes is greater than a first area threshold, retaining the plane with the higher position from the two horizontal planes to obtain an upper-layer plane; comparing each inclined plane pairwise, and when the distance between two inclined planes and the overlapping area of ​​the two inclined planes are greater than a second area threshold, retaining the plane with the higher position from the two inclined planes to obtain an upper-layer plane; comparing each horizontal plane with each inclined plane pairwise, and when the distance between a horizontal plane and an inclined plane is greater than or equal to a second distance threshold and the overlapping area of ​​a horizontal plane and an inclined plane is greater than a third area threshold, retaining the plane with the higher position from the horizontal plane and the inclined plane to obtain an upper-layer plane; and splicing the upper-layer planes to obtain the uppermost part plane.

[0045] Specifically, because 3D cameras may be positioned at different angles during shooting, not all planes will be horizontal; some may be inclined. Therefore, to more accurately determine the upper plane, the finely segmented planes are first compared with the calibration plate plane. Based on the angle between each finely segmented plane and the calibration plate plane, they are categorized as horizontal or inclined planes. The calibration plate plane is the surface of the steel plate part at the bottom of the part, placed beforehand for comparison with the plane. For example, if the part is placed in a corresponding pallet or basket in the storage unit, a calibration plate is placed on the bottom plate of the pallet or basket before placing the part. The plane captured by the 3D camera can then be compared with the calibration plate plane to determine whether it is horizontal or inclined. Then, the planes are grouped and compared in pairs according to horizontal planes, inclined planes and horizontal planes, and horizontal planes and inclined planes, and the upper plane is determined based on the corresponding conditions.

[0046] When comparing two horizontal planes pairwise, if the distance between two horizontal planes is less than a second distance threshold (e.g., less than one pixel), the angle difference between the two planes is calculated. If the angle difference is large, it indicates that the two horizontal planes belong to the same part. If not, i.e., the angle difference is less than the angle threshold, it indicates that the two horizontal planes do not belong to the same plane. In this case, the plane with the higher position can be selected and retained to obtain the upper plane. Simultaneously, the size of the overlapping area of ​​the two horizontal planes can be determined. If the overlapping area is greater than a preset first area threshold, the plane with the higher position is retained to obtain the upper plane.

[0047] When comparing inclined planes pairwise, if the distance between the two inclined planes is less than a threshold, and the area of ​​the overlapping region of the two inclined planes is greater than a preset second area threshold, the plane with the higher position is retained from the two inclined planes.

[0048] When comparing each horizontal plane with each inclined plane pairwise, if the distance between the horizontal plane and the inclined plane is less than 1 pixel, it indicates that they belong to the same part; conversely, if the distance is greater than or equal to 1 pixel, it indicates that they are not the same part. Further judgment is made based on the area of ​​the overlapping region between the horizontal plane and the inclined plane. That is, if the area of ​​the overlapping region between the horizontal plane and the inclined plane is greater than a preset third area threshold, the plane with the higher position is retained from the horizontal plane and the inclined plane. The second distance threshold, the first area threshold, the second area threshold, and the third area threshold are all preset values. The second distance threshold and the first distance threshold can be equal or unequal. The area thresholds can also be equal or unequal to each other.

[0049] The process of determining which plane is higher from two planes can be achieved by calculating the distance from a point to the plane. For example, if the distance between the point cloud closest to plane B in plane A and plane B is negative, it means plane A is below plane B; otherwise, plane A is above plane B. After obtaining the upper planes through pairwise comparisons, all the upper planes are stitched together to obtain the topmost part plane. Figure 5 The diagram shown provides a schematic representation of the plane of the topmost part. Figure 5 The image shown is from Figure 4The topmost part plane is extracted. Furthermore, due to the constantly changing stacking height of parts in reality, the part data captured by the camera may exhibit varying degrees of deformation. To avoid the impact of deformation, the stitching process can be further corrected using a calibration plate plane. This involves reprojecting the determined topmost plane onto the calibration plate plane to achieve accurate stitching. Specifically, a point cloud projection algorithm is used to project the point clouds corresponding to all extracted topmost planes onto the calibration plate plane to form a new point cloud. Then, using the intrinsic parameters of the 3D camera, the corresponding 2D mask image is calculated from the projected new point cloud. These mask images are then used for subsequent stitching. In the mask image, the point cloud portion and the non-point cloud portion are displayed in black and white respectively; for example, the point cloud portion is displayed in white, and the non-point cloud portion is displayed in black.

[0050] Step S105: Extract the topmost part from the stacked parts based on the plane of the topmost part.

[0051] Specifically, once the plane of the topmost part is extracted, the required topmost part can be accurately picked up from the stacked parts based on the plane of the topmost part and placed on the plate chain line. Subsequently, the topmost part is transported to the designated workstation for processing and use.

[0052] The above-described method for extracting the upper-layer parts of stacked components involves uniformly dividing a point cloud captured by a 3D camera into multiple small point cloud modules. Then, based on the mean square error, point cloud modules considered to belong to the same plane are integrated to obtain an initial segmentation plane. Further fine segmentation is performed based on the distance between the initial segmentation plane and adjacent point clouds at the plane boundary to obtain a more precise fine segmentation plane. Finally, by comparing pairs of fine segmentation planes, the upper-layer planes are selected and stitched together to obtain the upper-layer part plane for extraction. This method, based on 3D captured point clouds, achieves imaging of the uppermost part, featuring a large field of view, fast response speed, and wide environmental adaptability, remaining unaffected even in dark environments. Simultaneously, through point cloud merging, initial segmentation, and fine segmentation steps, the requirements for part stacking are reduced. Even with non-standard part stacking or workpiece bending deformation, the plane of the uppermost part can be accurately extracted and imaged, thereby improving the accuracy of subsequent automatic grasping.

[0053] In one embodiment, such as Figure 6 As shown, a device for extracting upper-layer parts of a stacked assembly is provided, comprising: Point cloud division module 601 is used to acquire the point cloud of the stacked components and divide the point cloud evenly into non-overlapping point cloud modules. The initial segmentation module 602 is used to merge each point cloud module into a plane based on the mean square error to obtain the initial segmentation plane; The fine segmentation module 603 is used to finely segment the initial segmentation plane based on the distance between the initial segmentation plane and the adjacent point clouds of the plane boundary, so as to obtain a fine segmentation plane; The plane extraction module 604 is used to compare and filter the upper planes of each finely segmented plane, and then splice the upper planes to obtain the uppermost part plane. The parts extraction module 605 extracts the topmost part from the stacked parts based on the plane of the topmost part.

[0054] In one embodiment, the upper part extraction device for the stacked components further includes a screening module for filtering out point cloud modules with missing data and discontinuous depth based on the depth value of the point cloud, and filtering out point cloud modules that do not meet the retention conditions based on the mean square error and normal vector of the point cloud modules.

[0055] In one embodiment, the filtering module is further configured to convert each point cloud in the point cloud module into a point cloud matrix; calculate the covariance matrix based on the point cloud matrix, obtain the eigenvalues ​​and eigenvectors of the covariance matrix, fit the plane equation of the point cloud in the point cloud module based on the eigenvalues ​​and eigenvectors; calculate the mean square error of each point cloud in the point cloud module based on the plane equation; and retain point cloud modules whose mean square error is less than a first error threshold.

[0056] In one embodiment, the filtering module is further configured to determine the normal vectors of each point cloud in the point cloud module; when the point cloud module is determined to include the boundary between the upper and lower planes based on the difference between the normal vectors, the point cloud module is filtered out.

[0057] In one embodiment, the initial segmentation module 602 is further configured to: determine the point cloud module with the smallest mean square error among the unmerged point cloud modules as the initial module; select point cloud modules from the adjacent point cloud modules of the initial module whose mean square error after merging with the initial module is less than a second error threshold, merge them to obtain a new initial module, and continuously select point cloud modules from the adjacent point cloud modules of the new initial module to continue merging until merging is no longer possible, thereby obtaining an initial segmentation plane; return to the step of determining the point cloud module with the smallest mean square error among the unmerged point cloud modules as the initial module, until all point cloud modules have been merged.

[0058] In one embodiment, the fine segmentation module 603 is further configured to determine the adjacent point clouds of each plane boundary of the initial segmentation plane, calculate the distance to the adjacent point clouds of each plane boundary according to the plane equation of each point cloud in the initial segmentation plane; when the distance is less than a first distance threshold, the adjacent point cloud of the plane boundary corresponding to the distance does not belong to other planes, and the mean square error of the plane after the adjacent point cloud of the plane boundary is included in the plane is less than a third error threshold, the adjacent point cloud of the plane boundary is included in the initial segmentation plane; when the distance is less than the first distance threshold, the adjacent point cloud of the plane boundary corresponding to the distance belongs to other planes, and the mean square error of the plane after the plane is merged is less than the third error threshold, the plane to which the adjacent point cloud of the plane boundary belongs is merged with the initial segmentation plane to obtain the fine segmentation plane.

[0059] In one embodiment, the plane extraction module 604 is further configured to determine horizontal planes and inclined planes based on the angles between each finely segmented plane and the calibration plate plane; compare each horizontal plane pairwise, and when the distance between two horizontal planes is less than a second distance threshold and the angle difference between two horizontal planes is less than an angle threshold, or the overlapping area of ​​two horizontal planes is greater than a first area threshold, retain the plane with the higher position from the two horizontal planes to obtain the upper plane; compare each inclined plane pairwise, and when the distance between two inclined planes and the overlapping area of ​​two inclined planes are greater than a second area threshold, retain the plane with the higher position from the two inclined planes to obtain the upper plane; compare each horizontal plane with each inclined plane pairwise, and when the distance between a horizontal plane and an inclined plane is greater than or equal to a second distance threshold and the overlapping area of ​​a horizontal plane and an inclined plane is greater than a third area threshold, retain the plane with the higher position from the horizontal plane and the inclined plane to obtain the upper plane; and splice the upper planes to obtain the uppermost part plane.

[0060] Specific limitations regarding the upper-layer component extraction device for stacked components can be found in the above-described limitations of the upper-layer component extraction method for stacked components, and will not be repeated here. Each module in the aforementioned upper-layer component extraction device for stacked components can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the embodiments of the upper-layer component extraction methods for stacked components described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms.

[0061] In one embodiment, a computer device is provided, which may be a server, including a processor, memory, and a network interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for extracting upper-layer components of a stacked assembly. Exemplarily, the computer program may be divided into one or more modules, one or more of which are stored in memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting various parts of the computer device via various interfaces and lines.

[0062] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0063] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for extracting the upper layer parts of a stacked assembly, characterized in that, include: Obtain the point cloud of the stacked components, and divide the point cloud evenly into non-overlapping point cloud modules; The process involves merging the point cloud modules into a plane based on their mean square error (MSE) to obtain an initial segmentation plane. This includes: identifying the point cloud module with the smallest MSE among the unmerged point cloud modules as the initial module; selecting point cloud modules from the adjacent point cloud modules of the initial module whose MSE after merging with the initial module is less than a second error threshold, merging them to obtain a new initial module, and continuously selecting point cloud modules from the adjacent point cloud modules of the new initial module to continue merging until merging is no longer possible, thus obtaining the initial segmentation plane; and returning to the step of identifying the point cloud module with the smallest MSE among the unmerged point cloud modules as the initial module, until all the point cloud modules have been merged. The initial segmentation plane is finely segmented based on the distance between the initial segmentation plane and the adjacent point clouds of the plane boundary to obtain a finely segmented plane. This includes: determining the adjacent point clouds of each plane boundary of the initial segmentation plane; calculating the distance to each adjacent point cloud of the plane boundary based on the plane equation of each point cloud in the initial segmentation plane; when the distance is less than a first distance threshold, the adjacent point cloud of the plane boundary corresponding to the distance does not belong to another plane, and the mean square error of the plane after the adjacent point cloud of the plane boundary is included in the plane is less than a third error threshold, the adjacent point cloud of the plane boundary is included in the initial segmentation plane; when the distance is less than the first distance threshold, the adjacent point cloud of the plane boundary corresponding to the distance belongs to another plane, and the mean square error of the plane after the plane is merged is less than the third error threshold, the plane to which the adjacent point cloud of the plane boundary belongs is merged with the initial segmentation plane to obtain a finely segmented plane. The process involves comparing each of the finely segmented planes pairwise to select an upper-level plane, and then stitching the upper-level planes together to obtain the uppermost part plane. This includes: determining horizontal and inclined planes based on the angles between each finely segmented plane and the calibration plate plane; comparing each horizontal plane pairwise, and if the distance between two horizontal planes is less than a second distance threshold and the angle difference between the two horizontal planes is less than an angle threshold, or the overlapping area of ​​the two horizontal planes is greater than a first area threshold, retaining the plane with the higher position from the two horizontal planes to obtain the upper-level plane; comparing each inclined plane pairwise, and if the distance between two inclined planes is less than a threshold and the overlapping area of ​​the two inclined planes is greater than a second area threshold, retaining the plane with the higher position from the two inclined planes to obtain the upper-level plane; comparing each horizontal plane and each inclined plane pairwise, and if the distance between the horizontal plane and the inclined plane is greater than or equal to a second distance threshold and the overlapping area of ​​the horizontal plane and the inclined plane is greater than a third area threshold, retaining the plane with the higher position from the horizontal plane and the inclined plane to obtain the upper-level plane; and stitching the upper-level planes together to obtain the uppermost part plane. The topmost part in the stack is extracted based on the plane of the topmost part.

2. The method according to claim 1, characterized in that, After obtaining the point cloud of the stacked components and uniformly dividing the point cloud into non-overlapping point cloud modules, the process further includes: Point cloud modules with missing data or discontinuous depth are filtered out based on their depth values, and point cloud modules that do not meet the retention criteria are filtered out based on their mean square error and normal vector.

3. The method according to claim 2, characterized in that, Point cloud modules that do not meet the retention criteria are filtered out based on their mean square error, including: Convert each point cloud in the point cloud module into a point cloud matrix; Calculate the covariance matrix based on the point cloud matrix, obtain the eigenvalues ​​and eigenvectors of the covariance matrix, and fit the plane equation of the point cloud in the point cloud module based on the eigenvalues ​​and eigenvectors. The mean square error of each point cloud in the point cloud module is calculated based on the plane equation. Point cloud modules whose mean square error is less than the first error threshold are retained.

4. The method according to claim 2, characterized in that, Point cloud modules that do not meet the retention criteria are filtered out based on their normal vectors, including: Determine the normal vector of each point cloud in the point cloud module; When the point cloud module is determined to include the boundary between the upper and lower planes based on the difference between the normal vectors, the point cloud module is filtered out.

5. A device for extracting upper-layer parts of a stacked assembly, characterized in that, The apparatus for implementing the method according to any one of claims 1-4, the apparatus comprising: A point cloud partitioning module is used to acquire the point cloud of the stacked components and uniformly divide the point cloud into non-overlapping point cloud modules. The initial segmentation module is used to merge the point cloud modules into a plane based on the mean square error to obtain the initial segmentation plane; The fine segmentation module is used to finely segment the initial segmentation plane based on the distance between the initial segmentation plane and the adjacent point clouds of the plane boundary, so as to obtain a fine segmentation plane; The plane extraction module is used to compare and filter the upper planes of each finely segmented plane in pairs, and to stitch the upper planes together to obtain the uppermost part plane. The parts extraction module extracts the topmost part from the stacked components based on the plane of the topmost part.

6. A computer device comprising a processor and a memory, the memory storing a computer program, characterized in that, The processor is used to execute the computer program to implement the method for extracting upper part of stacked components as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for extracting the upper part of the stacked component as described in any one of claims 1-4.

Images