Method, device and apparatus for unstacking

Abstract

This application provides a method, apparatus, and device for depalletizing, relating to the field of visual algorithm recognition. The method includes: acquiring actual three-dimensional point cloud data of the target material; extracting target point cloud data from the actual three-dimensional point cloud data based on the attribute information of the target material; converting the target point cloud data into a two-dimensional image; wherein the attribute information is the actual stacking parameters; determining the target corner points of the target material based on the two-dimensional image, and determining the corner point position and rotation angle; determining the grasping pose based on the corner point position and rotation angle, and sending the grasping pose to a preset robotic arm controller, so that the robotic arm controller performs a depalletizing operation on the material according to the grasping pose. This method aims to improve the positioning accuracy of depalletizing.

Description

Technical Field

[0001] This application relates to the field of visual algorithm recognition, and more specifically, to a method, apparatus, and device for depalletizing and unpalletizing. Background Technology

[0002] Currently, with the accelerated pace of transformation and upgrading in the manufacturing industry, the traditional manual depalletizing and palletizing operation mode can no longer meet the needs of modern production. In industries such as pharmaceuticals, food, electronics, and e-commerce logistics, depalletizing and palletizing operations are often characterized by high repetition and labor intensity, becoming a key bottleneck restricting the improvement of enterprise production efficiency. According to industry survey data, on a typical packaging production line, manual depalletizing and palletizing accounts for about 30% of the total working hours, and the problems of quality fluctuations and efficiency instability caused by human factors are becoming increasingly prominent. This mainly includes three challenges: labor cost pressure, as the depalletizing and palletizing job is monotonous and tedious, resulting in high staff turnover and continuously rising recruitment and training costs. Furthermore, it faces the problem of operator fatigue and low efficiency; production efficiency bottleneck, as the speed of manual depalletizing and palletizing is limited and cannot match the high-speed cycle requirements of modern production lines, making the traditional manual mode a key factor restricting capacity release; and quality consistency challenges, as manual operation cannot guarantee the stability and accuracy of pallet arrangement. In industries such as medical and precision electronics, improper stacking may lead to product damage or safety hazards during transportation.

[0003] In the existing technology, although automated equipment is regarded as the best alternative, the traditional depalletizing and palletizing system currently faces two major technical challenges in the operation scenario: First, the existing vision modality is difficult to meet the requirements of large field of view coverage and millimeter-level accuracy; second, the stability of corner feature extraction in dynamic environment is insufficient, especially for complex working conditions such as pallet wear and deformation, and metal packaging reflection, the feature false detection rate is as high as 15%.

[0004] Therefore, there is an urgent need for a method to improve the positioning accuracy of palletizing and unpacking. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, and device for depalletizing and palletizing, which solves the above-mentioned problems existing in the prior art and can improve the positioning accuracy of depalletizing and palletizing.

[0006] Firstly, a method for depalletizing and unpalletizing is provided, which may include:

[0007] Acquire the actual 3D point cloud data of the target material;

[0008] Based on the attribute information of the target material, target point cloud data is extracted from the actual three-dimensional point cloud data; the target point cloud data is converted into a two-dimensional image; wherein, the attribute information is the actual stacking parameters;

[0009] Based on the two-dimensional image, the target corner point of the target material is determined, and the corner point position and rotation angle are determined. Based on the corner point position and the rotation angle, the gripping posture is determined, and the gripping posture is sent to a preset robotic arm controller so that the robotic arm controller performs a depalletizing operation on the target material according to the gripping posture.

[0010] Secondly, a depalletizing device is provided, which may include:

[0011] The acquisition module is used to acquire the actual 3D point cloud data of the target material;

[0012] The interception module is used to intercept target point cloud data from the actual three-dimensional point cloud data based on the attribute information of the target material.

[0013] A conversion module is used to convert the target point cloud data into a two-dimensional image; wherein the attribute information is the actual stacking parameters;

[0014] The execution module is used to determine the target corner of the target material based on the two-dimensional image, and to determine the corner position and rotation angle of the target corner; to determine the gripping posture based on the corner position and the rotation angle, and to send the gripping posture to a preset robotic arm controller, so that the robotic arm controller performs a depalletizing operation on the target material based on the gripping posture.

[0015] Thirdly, an electronic device is provided, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0016] Memory, used to store computer programs;

[0017] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect above.

[0018] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of any of the methods described in the first aspect above.

[0019] This application provides a depalletizing method, apparatus, and device that acquires actual three-dimensional point cloud data of the target material. Based on the target material's attribute information, target point cloud data is extracted from the actual three-dimensional point cloud data; the target point cloud data is converted into a two-dimensional image; wherein the attribute information represents actual stacking parameters. Based on the two-dimensional image, target corner points of the target material are determined, along with their corner positions and rotation angles; based on the corner positions and rotation angles, a gripping pose is determined and sent to a preset robotic arm controller, enabling the robotic arm controller to perform depalletizing operations on the material according to the gripping pose. This solution addresses the problem of insufficient positioning accuracy (±5 mm) of traditional depalletizing visual algorithms by innovatively using a multimodal algorithm for depalletizing positioning. Specifically, it utilizes a hybrid architecture combining 3D and 2D vision algorithms to accurately identify target corner points, achieving precise positioning of pallet corner points and pallet rotation angles, and generating a gripping pose used to execute the depalletizing operation. Therefore, using 3D combined with 2D vision algorithms greatly improves algorithm efficiency and effectively solves the problem of low positioning accuracy caused by various interferences near the top layer of materials being depalletized in actual depalletizing. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a depalletizing method provided in an embodiment of this application;

[0022] Figure 2 A flowchart illustrating a destacking and palletizing method provided in this application;

[0023] Figure 3 A schematic diagram illustrating a scenario for a depalletizing method provided in this application;

[0024] Figure 4 A schematic diagram illustrating another scenario for depalletizing / unpalletizing methods provided in this application;

[0025] Figure 5 A schematic diagram illustrating another scenario for depalletizing / unpalletizing methods provided in this application;

[0026] Figure 6 A schematic diagram illustrating another scenario for depalletizing / unpalletizing methods provided in this application;

[0027] Figure 7 This is a schematic diagram of the structure of a depalletizing device provided in an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by those skilled in the art. The words "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are only used to distinguish different components. The words "comprising" or "including," etc., mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, but do not exclude other elements or objects. The words "connected," "coupled," or "connected," etc., are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up," "down," "left," "right," etc., are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0030] The depalletizing method provided in this application can be applied to electronic devices, terminal devices, depalletizing devices or equipment, or other devices or equipment that can execute this embodiment, and there are no limitations on this.

[0031] Terminal equipment can be user equipment (UE) such as mobile phones, smartphones, laptops, digital broadcast receivers, personal digital assistants (PDAs), and tablet computers (PADs), handheld devices, in-vehicle devices, wearable devices, computing devices or other processing devices connected to a wireless modem, mobile stations (MS), mobile terminals, etc. This terminal has the ability to communicate with one or more core networks via a radio access network (RAN).

[0032] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0033] Figure 1 This is a flowchart illustrating a depalletizing and depalletizing method provided in an embodiment of this application. Figure 1 As shown, the method may include:

[0034] Step S101: Obtain the actual three-dimensional point cloud data of the target material.

[0035] For example, a depth map can be obtained by capturing the target material with a 3D camera or other acquisition device. Then, the pixel coordinates (u,v) and depth value D(u,v) in the depth map can be transformed by inverse projection using the intrinsic parameter matrix of the 3D camera to convert the pixel coordinates (u,v) and depth value D(u,v) in the depth map into three-dimensional (3D) point cloud coordinates (Xc,Yc,Zc) in the camera coordinate system.

[0036] Optionally, the actual 3D point cloud data can be divided into a uniform grid, with a representative point randomly selected within each grid. The advantage of this algorithm is its computational complexity of O(n), allowing for rapid initial filtering and simplification of the point cloud. Further, voxel downsampling is used to perform a second downsampling on the filtered point cloud, further refining the data while preserving geometric features (such as edges and corners), outputting a lightweight point cloud that retains key features. This layered processing balances speed and accuracy requirements, enabling rapid downsampling while maximizing the preservation of material point cloud features.

[0037] Step S102: Based on the attribute information of the target material, extract the target point cloud data from the actual three-dimensional point cloud data; convert the target point cloud data into a two-dimensional image; wherein, the attribute information is the actual stacking parameters.

[0038] For example, the attribute information of the target material is the actual stacking parameters, such as the length, width, and height of the target material, the material type, and the physical properties of the material, etc., which are not limited. Based on the actual stacking parameters of the target material, the target intercept thickness is determined, and target point cloud data is extracted from the actual 3D point cloud data according to the target intercept thickness. Then, using an orthographic projection algorithm for the point cloud data, the 3D point cloud in the target point cloud data is orthographically projected (top view) onto a 2D image plane to obtain a converted 2D image. For example, through scaling and offset calculations, the 3D point cloud in the target point cloud data is mapped to the pixel positions of the 2D image to generate a binarized image, i.e., a 2D image.

[0039] Step S103: Based on the two-dimensional image, determine the target corner point of the target material, and determine the corner point position and rotation angle; based on the corner point position and rotation angle, determine the gripping posture, and send the gripping posture to the preset robotic arm controller, so that the robotic arm controller can perform depalletizing operation on the material according to the gripping posture.

[0040] For example, a two-dimensional image is analyzed using a preset principal component analysis method to obtain at least one candidate corner point. The target corner point of the target material is then identified from these candidate corner points, and its position and rotation angle are determined. Finally, based on the corner point position and rotation angle, a gripping pose is determined and sent to a preset robotic arm controller. The robotic arm controller receives the gripping pose and performs a depalletizing / palletizing operation on the material according to the gripping pose.

[0041] The method provided in this application embodiment acquires the actual three-dimensional point cloud data of the target material. Based on the attribute information of the target material, target point cloud data is extracted from the actual three-dimensional point cloud data; the target point cloud data is converted into a two-dimensional image; wherein the attribute information is the actual stacking parameters. Based on the two-dimensional image, the target corner points of the target material are determined, and the corner point position and rotation angle are determined; based on the corner point position and rotation angle, the grasping pose is determined and sent to a preset robotic arm controller, so that the robotic arm controller performs a depalletizing operation on the material according to the grasping pose. In this solution, addressing the problem of insufficient positioning accuracy (±5 mm) of traditional depalletizing visual algorithms, a multimodal algorithm is innovatively used for depalletizing positioning. That is, a hybrid architecture is processed using a combination of 3D and 2D vision algorithms to accurately identify target corner points, achieve precise positioning of pallet corner points and pallet rotation angles, and generate a grasping pose, which is used to execute the depalletizing operation. Therefore, using a combination of 3D and 2D vision algorithms greatly improves algorithm efficiency and effectively solves the problem of low depalletizing positioning accuracy caused by various interferences near the top layer of material in actual depalletizing.

[0042] Figure 2 A flowchart illustrating a depalletizing and palletizing method provided in this application is shown below. Figure 2 As shown, in this embodiment... Figure 1 Based on the embodiments, the method is described in detail below, and the method includes:

[0043] Step S201: Obtain the actual three-dimensional point cloud data of the target material.

[0044] For example, this step is described in step S101, and will not be repeated here.

[0045] Step S202: Based on the attribute information of the target material, extract the target point cloud data from the actual 3D point cloud data.

[0046] In one example, S202 includes: analyzing the distribution of actual 3D point cloud data in the preset Z-axis direction, determining the top surface of the material, and obtaining the height reference value corresponding to the top surface; determining the interception coefficient based on the attribute information of the target material; determining the target interception thickness based on the preset reference thickness and interception coefficient; determining the difference between the height reference value and the target interception thickness, and intercepting the target point cloud data in the actual 3D point cloud data between the difference and the height reference value.

[0047] For example, after filtering out redundant outliers using a preset radius filter, the point cloud of the top layer of material is dynamically cropped based on the current material height and thickness to prevent the influence of lower layers. Specifically, an outlier identification and removal algorithm is used to analyze the point density within a preset spherical neighborhood around each point using a radius filter. First, a spherical region with a preset radius is defined for each point in the point cloud, and the number of neighboring points within this region is counted. If the number of neighboring points around a point is less than a preset threshold, it is determined to be a noise point and deleted. This processing method can effectively remove individual outliers while maintaining the integrity of the main structure of the point cloud.

[0048] After removing outliers, statistical methods are used to analyze the distribution of the actual 3D point cloud data along the preset Z-axis to determine the top surface of the material and calculate the corresponding height reference value. The densest point cloud layer detected at the top surface (i.e., the "top surface") may contain a small number of point clouds above it (closer to the positive Z-axis direction). The height reference value is the height of the topmost "top surface" of the target material. Material attribute information includes the length, width, and height of the material stack, material type, and physical properties. Based on this attribute information, a truncation coefficient is dynamically adjusted in real time. Then, based on the preset reference thickness and truncation coefficient, the target truncation thickness is dynamically determined: target truncation thickness = preset reference thickness × dynamically determined truncation coefficient. Finally, the difference between the height reference value and the target truncation thickness is determined, and a pass-through filter is used to extract and retain the target point cloud data between the difference and the height reference value, effectively isolating interference from lower material layers.

[0049] Therefore, dynamic interception is based on the detected top surface of the top layer, and a dynamic target interception thickness is taken downward (in the negative direction of the Z-axis), so as to extract the point cloud located between the difference and the height reference value in physical space, which corresponds exactly to the material entity of the top layer.

[0050] Step S203: Convert the target point cloud data into a two-dimensional image; wherein the attribute information is the actual stacking parameters.

[0051] In one example, S203 includes: determining the boundary range and actual scaling ratio of the target point cloud data; generating an image matrix based on the boundary range and actual scaling ratio; performing orthophoto projection on the target point cloud data to map the three-dimensional points in the target point cloud data to the pixel positions in the image matrix, thereby generating a two-dimensional image.

[0052] For example, the conversion of target point cloud data to a 2D image uses an orthophoto projection algorithm to project the 3D point cloud onto a 2D image plane. Specifically, firstly, based on a preset PCA algorithm, the boundary range of the target point cloud data, and the actual scaling ratio, an image matrix for the output image is calculated. The image matrix includes the number of rows and columns, including reserved space for the border size. During the projection process, each point in the target point cloud data is traversed according to the orthophoto projection algorithm. Through scaling and offset calculations, each point is mapped to a specific row and column pixel position in the image matrix, and the RGB channels of the pixels at each pixel position are set to 255. This generates a top-view projection of the point cloud onto the 2D image, i.e., a 2D image, providing input data for subsequent contour extraction and corner detection. The RGB values ​​used are merely an example and are not intended to be limiting.

[0053] Step S204: Extract the outermost contour point set of the target material in the two-dimensional image.

[0054] For example, according to a preset arc detection algorithm, the outermost contour point set of the target material is extracted in the two-dimensional image, and the contour point set is the edge contour of the target material.

[0055] Step S205: Perform principal component analysis on the contour point set to generate candidate corner points and candidate positions of the candidate corner points.

[0056] In one example, S205 includes: calculating the covariance matrix of the contour point set based on a preset principal component analysis method, and performing eigenvalue decomposition on the covariance matrix to obtain the first eigenvalue and the second eigenvalue; generating the first eigenvector corresponding to the first eigenvalue and the second eigenvector corresponding to the second eigenvalue; determining the first eigenvector and the second eigenvector as the principal direction of the contour point set; determining the center point of the contour point set, and establishing a target coordinate system based on the center point; wherein the X-axis of the target coordinate system points to the direction of the first eigenvector, and the Y-axis of the target coordinate system points to the direction of the second eigenvector; transforming the coordinates of each contour point in the contour point set to the target coordinate system; for each contour point in the target coordinate system, determining the projection score of the contour point according to a preset weighted average strategy, and generating candidate corner points and candidate positions of the candidate corner points based on the projection score.

[0057] For example, based on a preset principal component analysis (PCA) method, the extracted contour point set is analyzed and calculated to obtain the covariance matrix of the gradient within the neighborhood of the contour point set. By performing eigenvalue decomposition on the covariance matrix, a first eigenvalue λ1 and a second eigenvalue λ2 are obtained, generating a first eigenvector corresponding to the first eigenvalue λ1 and a second eigenvector corresponding to the second eigenvalue λ2. Using the first and second eigenvectors, the principal direction of the material is determined as the directions pointed to by the first and second eigenvectors, respectively. The center point of the contour point set is determined, and a standardized target coordinate system (i.e., the PCA coordinate system) is established based on the center point. The X-axis of the target coordinate system points to the direction of the first eigenvector, and the Y-axis points to the direction of the second eigenvector, thus ensuring that the first principal component (i.e., the X-axis) points to the right and the second principal component (i.e., the Y-axis) points upward. In the PCA coordinate system, the projected coordinates of each contour point are calculated according to the algorithm, thereby transforming the coordinates of all contour points in the contour point set to the target coordinate system. Finally, for each contour point in the target coordinate system, a pre-defined weighted averaging strategy is used to determine the projection score of the contour point. The probability of a corner point is evaluated based on the projection score, and multiple candidate corner points are generated. After obtaining the candidate corner points, when the projection scores of multiple contour points are close, a secondary verification is performed using a linear combination of the original coordinates. The point with the highest projection score is ultimately determined as the precise candidate corner point, and its candidate position is also determined. The determination of the projection score of the contour points may utilize the distances between each contour point and the center point; alternatively, it may utilize the distances between each contour point and the center point, as well as the extreme projection values ​​of each contour point on the X and Y axes, etc., without limitation.

[0058] Step S206: Based on the preset minimum enclosing circle algorithm and candidate positions, determine the target corner point of the target material, and determine the corner point position and rotation angle of the target corner point.

[0059] In one example, S206 includes: collecting a set of arc feature points from a contour point set within a preset range, centered on the candidate location; wherein the set of arc feature points includes arc feature points used to describe the arc shape of the target material; fitting a minimum area circle containing the arc feature points according to a preset minimum bounding circle algorithm, and generating the starting angle and ending angle of the arc shape based on the minimum area circle; calculating two intersection points between the arc shape and the contour point set based on the starting angle and ending angle of the arc shape; extending along the tangent direction of the contour point set based on the intersection points, and fitting the extension lines corresponding to each intersection point according to a preset least squares method; determining the intersection point of the extension lines as the target corner point, and determining the corner point position of the target corner point; determining the direction vector of each extension line, and calculating the angle between the direction vectors according to the preset geometric constraints corresponding to the arc shape; the angle is the rotation angle.

[0060] In one example, "fitting a minimum area circle containing arc feature points according to a preset minimum bounding circle algorithm, and generating the starting and ending angles of the arc shape based on the minimum area circle" includes: fitting a minimum area circle containing arc feature points according to a preset minimum bounding circle algorithm, obtaining the center, center coordinates, and radius parameters of the minimum area circle; calculating the vector angle of each arc feature point relative to the center based on the center coordinates and radius parameters; sorting the vector angles to generate a vector angle sequence; wherein the vector angle sequence includes the starting angle and the ending angle; and normalizing the starting angle and the ending angle to generate normalized starting angle and ending angle.

[0061] For example, a set of arc feature points is collected within a preset range, centered on the candidate location. This set includes arc feature points describing the arc shape of the target material; the arc shape can be an arc at the corner of the target material. Using a preset minimum bounding circle algorithm, these arc feature points are processed to fit a minimum area circle containing all arc feature points, obtaining key geometric features such as the center, center coordinates, and radius parameters of the minimum area circle. Based on the center coordinates and radius parameters, the vector angle of each arc feature point relative to the center is calculated, and the vector angles are sorted to generate a vector angle sequence. This sequence includes a starting angle (first position) and a ending angle (last position), thus completing the accurate description of the arc feature. The starting and ending angles are then normalized to generate normalized starting and ending angles.

[0062] Then, based on the starting and ending angles of the arc shape, the two intersection points of the fitted arc shape and the contour point set are calculated. Extending from the intersection points along the tangent direction of the contour point set, suitable extension points are selected to construct straight line segments. Specifically, contour points are collected on both sides of the candidate corner point, and two extended lines are fitted using the least squares method to determine the intersection point of the two extended lines. This intersection point is the precise target corner point, and its position is determined. Finally, using the direction vectors of the two fitted extended lines and according to the preset geometric constraints corresponding to the arc shape, the included angle is calculated using the vector dot product formula. This included angle is the accurately calculated overall rotation angle.

[0063] The preset geometric constraints can include constraints for intersection point positioning, constraints for straight line direction, and constraints for verification and correction of angle calculation. The constraint for intersection point positioning means that the sampling starting point (i.e., the two intersection points) of the subsequent straight line fitting must be located at the theoretical transition point between the arc and the straight line, thereby ensuring that the fitted straight line is the true straight edge of the material, rather than erroneous data containing arc segments, thus providing a "correct starting point" for straight line fitting. The constraint for straight line direction refers to the known tangent direction, which can be used as guidance or verification. The algorithm can collect contour points in a small neighborhood along the tangent direction to ensure that the collected points are indeed extending along the straight edge, eliminating the possibility of collecting points in the wrong direction due to contour noise or defects. The verification and correction constraints for angle calculation refer to the angle between two fitted lines calculated directly through the vector dot product. This angle can be compared with the vector angle derived from the geometric relationship of the center of the circle. If the deviation between the two is within the allowable error, the result is reliable. If a slight deviation in the fitting of a line is caused by local contour defects, stronger global geometric constraints such as "the two opposite lines should be parallel" and "the line connecting the centers of the circle indicates the direction" can be used to optimize or weight the fitting result, thereby obtaining a more stable and accurate final rotation angle.

[0064] Step S207: Determine the gripping pose based on the corner position and rotation angle, and send the gripping pose to the preset robotic arm controller so that the robotic arm controller can perform depalletizing and palletizing operations on the material according to the gripping pose.

[0065] In one example, the attribute information includes the length and width dimensions of the target material; S207 includes: determining the center coordinates of the target material based on the corner positions and the length and width dimensions; determining the gripping pose based on the center coordinates, rotation angle, and preset gripping height, and sending the gripping pose to the preset robotic arm controller.

[0066] For example, the attribute information of the target material includes the length, width, and height of the stacked material, material type, and physical properties. The center coordinates of the target material can be inferred from its geometric model (e.g., a rectangle defined by its length, width, and height) and the located target corner points. Based on the center coordinates, corner point positions, rotation angle, and a preset gripping height (Z value), the gripping pose (X, Y, Z, Roll, Pitch, Yaw) is determined and sent to a preset robotic arm controller. The robotic arm controller receives the gripping pose and performs a depalletizing / unpalletizing operation on the material according to the gripping pose.

[0067] The method provided in this application embodiment acquires actual three-dimensional point cloud data of a target material. Based on the attribute information of the target material, target point cloud data is extracted from the actual three-dimensional point cloud data. The target point cloud data is converted into a two-dimensional image; wherein the attribute information is the actual stacking parameters. The outermost contour point set of the target material in the two-dimensional image is extracted. Principal component analysis is performed on the contour point set to generate candidate corner points and candidate positions of the candidate corner points. Based on a preset minimum bounding circle algorithm and the candidate positions, the target corner points of the target material are determined, along with their corner point positions and rotation angles. Based on the corner point positions and rotation angles, a grasping pose is determined and sent to a preset robotic arm controller, so that the robotic arm controller performs a depalletizing and palletizing operation on the material according to the grasping pose. In this solution, to address the problem of insufficient positioning accuracy (±5 mm) of traditional palletizing visual algorithms, a multimodal algorithm is innovatively used for palletizing positioning, achieving an accuracy of ±1 mm. The solution utilizes a hybrid architecture that combines 3D and 2D vision algorithms. A 3D camera is used to quickly construct the pallet corner point cloud topology. Combined with a corner detection algorithm, the solution achieves accurate positioning of the pallet target corners and the pallet rotation angle. Furthermore, the solution combines the pallet's length and width to infer the pallet center coordinates and generate a grasping pose, which is used to perform the palletizing operation. Therefore, using 3D combined with 2D vision algorithms greatly improves algorithm efficiency. Among them, algorithms such as PCA (i.e., determining the target coordinate system based on actual 3D point cloud data), arc detection (i.e., including collecting arc feature point sets, performing circle fitting using a preset minimum bounding circle algorithm, arc judgment and analysis, and outputting arcs), and dynamic calculation of edge points (i.e., including collecting arc feature points near candidate corner points, extending along the contour tangent direction from the intersection point and selecting appropriate extension points to construct straight line segments) also greatly improve material positioning accuracy. This effectively solves the problem of low positioning accuracy caused by various interferences near the top layer of materials in actual depalletizing.

[0068] In one example Figure 3 This application provides a scenario illustration of a depalletizing / palletizing method, as shown in the diagram. Figure 3 As shown, Figure 3 This refers to a 2D image captured and converted using a 3D camera, with the material in the center of the image.

[0069] In one example Figure 4 A schematic diagram illustrating another scenario for depalletizing / palletizing methods provided in this application, such as... Figure 4 As shown, this includes a top-down view of the point cloud. Specifically, the actual 3D point cloud data acquired by the 3D camera is downsampled, the target 3D point cloud of the topmost material is extracted, and the target 3D point cloud is converted into a 2D image, which is the top-down view of the point cloud.

[0070] In one example Figure 5 A schematic diagram illustrating another scenario for depalletizing / palletizing methods provided in this application, such as... Figure 5 As shown, this includes a 2D image, thick solid line 1, and thick solid line 2. Using principal component analysis and arc detection algorithms, the coarse positions of candidate corner points of the material are obtained. The intersection of thick solid lines 1 and 2 is the position of the candidate corner point, and thick solid lines 1 and 2 are also the extension lines of the adjacent sides of the candidate corner point.

[0071] In one example Figure 6 A schematic diagram illustrating another scenario for depalletizing / palletizing methods provided in this application, such as... Figure 6 As shown, the material's outline (i.e. Figure 5 The unthickened lines (excluding the thickened solid lines 1 and 2) are compared with the fitted arc shape located in the ring. The two intersection points and their extension points are used to fit a straight line, which is represented by the thickened dashed lines 3 and 4. Then, the target corner point is precisely located. The rotation angle of the material is calculated based on the extension line of the adjacent side. The center coordinates of the material are inferred by combining the target corner point with the length and width of the material. The gripping posture of the robotic arm controller is corrected based on the center coordinates and rotation angle.

[0072] Corresponding to the above method, embodiments of this application also provide a depalletizing device, such as... Figure 7 As shown, it includes:

[0073] Acquisition module 41 is used to acquire the actual three-dimensional point cloud data of the target material;

[0074] The interception module 42 is used to intercept target point cloud data in the actual three-dimensional point cloud data according to the attribute information of the target material;

[0075] The conversion module 43 is used to convert the target point cloud data into a two-dimensional image; wherein the attribute information is the actual stacking parameters;

[0076] The execution module 44 is used to determine the target corner of the target material based on the two-dimensional image, and to determine the corner position and rotation angle of the target corner; to determine the gripping posture based on the corner position and the rotation angle, and to send the gripping posture to a preset robotic arm controller, so that the robotic arm controller performs a depalletizing operation on the target material based on the gripping posture.

[0077] The functions of each functional unit of the depalletizing and palletizing device provided in the above embodiments of this application can be implemented through the above methods and steps. Therefore, the specific working process and beneficial effects of each unit in the depalletizing and palletizing device provided in the embodiments of this application will not be repeated here.

[0078] This application also provides an electronic device, such as... Figure 8As shown, it includes a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540.

[0079] Memory 530 is used to store computer programs;

[0080] The processor 510 performs the above steps when executing the program stored in the memory 530.

[0081] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0082] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0083] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0084] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0085] The implementation methods and beneficial effects of the various components of the electronic device in the above embodiments for solving the problem can be found in [reference needed]. Figure 1 The steps in the illustrated embodiments are used to implement the electronic device. Therefore, the specific working process and beneficial effects of the electronic device provided in this application will not be repeated here.

[0086] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the depalletizing methods described in the above embodiments.

[0087] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the depalletizing methods described in the above embodiments.

[0088] Those skilled in the art will understand that the embodiments in this application can be provided as methods, systems, or computer program products. Therefore, the embodiments in this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments in this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] Although preferred embodiments have been described in this application, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0093] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims in this application and their equivalents, then this application also intends to include these modifications and variations.

Claims

1. A method for destacking and stacking, characterized in that, The method includes: Acquire the actual 3D point cloud data of the target material; Based on the attribute information of the target material, target point cloud data is extracted from the actual three-dimensional point cloud data; the target point cloud data is converted into a two-dimensional image; wherein, the attribute information is the actual stacking parameters; Based on the two-dimensional image, the target corner point of the target material is determined, and the corner point position and rotation angle are determined; based on the corner point position and the rotation angle, the grasping posture is determined, and the grasping posture is sent to the preset robotic arm controller, so that the robotic arm controller performs a depalletizing operation on the target material according to the grasping posture; Based on the two-dimensional image, the target corner point is determined, and the corner point position and rotation angle of the target corner point are determined, including: Extract the outermost contour point set of the target material from the two-dimensional image; Principal component analysis is performed on the contour point set to generate candidate corner points and candidate positions of the candidate corner points; Based on the preset minimum enclosing circle algorithm and the candidate positions, the target corner point of the target material is determined, and the corner point position and rotation angle of the target corner point are determined. Based on a preset minimum bounding circle algorithm and the candidate positions of the candidate corner points, the target corner points of the target material are determined, and the corner point position and rotation angle of the target corner points are determined, including: Centered on the candidate position, a set of arc feature points is collected within a preset range from the contour point set; wherein, the set of arc feature points includes arc feature points used to describe the arc shape of the target material; According to the preset minimum bounding circle algorithm, the minimum area circle containing the arc feature points is fitted to obtain the minimum area circle, and the starting angle and ending angle of the arc shape are generated according to the minimum area circle. Based on the starting and ending angles of the arc shape, calculate the two intersection points between the arc shape and the set of contour points; based on the intersection points, extend along the tangent direction of the set of contour points, and fit the extension lines corresponding to each intersection point according to the preset least squares method. The intersection of the extended lines is determined as the target corner point, and the corner point position of the target corner point is determined; Determine the direction vector of each extension line, and calculate the angle between the direction vectors according to the preset geometric constraints corresponding to the arc shape; the angle is the rotation angle.

2. The method as described in claim 1, characterized in that, Based on the attribute information of the target material, target point cloud data is extracted from the actual 3D point cloud data, including: The distribution of the actual three-dimensional point cloud data in the preset Z-axis direction is analyzed to determine the top surface of the material and obtain the height reference value corresponding to the top surface of the material. Based on the attribute information of the target material, determine the interception coefficient; The target cutting thickness is determined based on the preset reference thickness and the cutting coefficient. Determine the difference between the height reference value and the target intercept thickness, and intercept the target point cloud data from the actual three-dimensional point cloud data between the difference and the height reference value.

3. The method as described in claim 1, characterized in that, Converting the target point cloud data into a two-dimensional image includes: Determine the boundary range and actual scaling ratio of the target point cloud data; Generate an image matrix based on the boundary range and the actual scaling ratio; The target point cloud data is subjected to orthophoto projection processing to map the three-dimensional points in the target point cloud data to the pixel positions in the image matrix, thereby generating a two-dimensional image.

4. The method as described in claim 1, characterized in that, Principal component analysis is performed on the contour point set to generate candidate corner points and candidate positions of the candidate corner points, including: Based on the preset principal component analysis method, the covariance matrix of the contour point set is calculated, and the covariance matrix is ​​subjected to eigenvalue decomposition to obtain the first eigenvalue and the second eigenvalue. Generate a first feature vector corresponding to the first feature value and a second feature vector corresponding to the second feature value; Determine the first feature vector and the second feature vector as the main direction of the contour point set; Determine the center point of the contour point set, and establish a target coordinate system based on the center point; wherein the X-axis of the target coordinate system points to the direction of the first feature vector, and the Y-axis of the target coordinate system points to the direction of the second feature vector; The coordinates of each contour point in the contour point set are transformed into the target coordinate system. For each contour point in the target coordinate system, the projection score of the contour point is determined according to a preset weighted average strategy, and candidate corner points and candidate positions of the candidate corner points are generated based on the projection score.

5. The method as described in claim 1, characterized in that, Based on a preset minimum bounding circle algorithm, a minimum area circle containing the arc feature points is fitted, and based on the minimum area circle, the starting angle and ending angle of the arc shape are generated, including: Based on the preset minimum bounding circle algorithm, the minimum area circle containing the arc feature points is fitted to obtain the center, center coordinates and radius parameters of the minimum area circle; Based on the center coordinates and the radius parameter, calculate the vector angle of each arc feature point relative to the center of the circle; The vector angles are sorted to generate a vector angle sequence; wherein the vector angle sequence includes a start angle and an end angle; The starting angle and the ending angle are normalized to generate normalized starting angle and ending angle.

6. The method according to any one of claims 1-5, characterized in that, The attribute information includes the length and width dimensions of the target material; based on the corner positions and the rotation angle, the grasping posture is determined and sent to a preset robotic arm controller, including: Based on the corner positions and the length and width dimensions, determine the center coordinates of the target material; Based on the center coordinates, the rotation angle, and the preset gripping height, the gripping pose is determined, and the gripping pose is sent to the preset robotic arm controller.

7. A depalletizing and palletizing device for implementing the method of claim 1, characterized in that, The device includes: The acquisition module is used to acquire the actual 3D point cloud data of the target material; The interception module is used to intercept target point cloud data from the actual three-dimensional point cloud data based on the attribute information of the target material. A conversion module is used to convert the target point cloud data into a two-dimensional image; wherein the attribute information is the actual stacking parameters; The execution module is used to determine the target corner of the target material based on the two-dimensional image, and to determine the corner position and rotation angle of the target corner; to determine the gripping posture based on the corner position and the rotation angle, and to send the gripping posture to a preset robotic arm controller, so that the robotic arm controller performs a depalletizing operation on the target material based on the gripping posture.

8. An electronic device, characterized in that, The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-6.

Images