An object stacking method, system, device and product based on visual perception

Abstract

The application discloses a vision perception-based object stacking method, system, device and product, and relates to the field of logistics automation.The method comprises generating a second three-dimensional point cloud of a target object in a world coordinate; projecting the second three-dimensional point cloud along three coordinate axes to obtain three two-dimensional density heat maps; calculating the global density of each two-dimensional density heat map; setting a circle with a preset radius as the center of each two-dimensional point, and calculating the local density in each circle; if the local density in the circle is less than the global density, the corresponding two-dimensional point is removed from the two-dimensional density heat map; the maximum two-dimensional connected region in each two-dimensional density heat map is calculated, and the three-dimensional points in the second three-dimensional point cloud that are projected and fall in the three maximum two-dimensional connected regions are retained to form a target three-dimensional point cloud; and according to the target placement region of the target object in the stacking region, the application can accurately calculate the size of the object to facilitate accurate execution of the stacking operation.

Description

Technical Field

[0001] This application relates to the field of logistics automation, and in particular to a visual perception-based method, system, device, and product for palletizing objects. Background Technology

[0002] Against the backdrop of rapid automation in the logistics industry, intelligent palletizing systems, as a core component of modern warehousing and logistics, face new opportunities and challenges in technological innovation. A complete palletizing process requires the coordinated operation of three key stages: acquiring the three-dimensional pose information of the object at the end of the conveyor belt, accurately measuring the object's dimensions, and finally dynamically calculating the optimal placement position based on the object's dimensions. However, existing palletizing methods typically only implement one or two of these stages. While combining different palletizing methods can form a complete palletizing process, poor coordination or incompatibility leads to low palletizing efficiency.

[0003] More importantly, in the process of measuring object dimensions, it is necessary to construct an effective 3D point cloud of the object based on its RGB-D image. However, in addition to the target object, the image also includes background objects such as conveyor belts. Therefore, the preliminary 3D point cloud extracted from the RGB-D image contains a large number of noise points that need to be removed. The traditional approach is to set the height of the conveyor belt as a threshold, and then directly remove 3D points with a height less than the threshold from the preliminary 3D point cloud. The remaining noise points are then removed using the DBSCAN algorithm. This 3D point cloud construction method has at least two drawbacks:

[0004] 1. Due to vibration and wear, conveyor belts exhibit non-uniform height deviations, meaning their surfaces are not perfectly horizontal. Furthermore, objects are tightly placed on the conveyor belt surface. Therefore, simply using a height threshold to remove noise points belonging to the conveyor belt surface not only results in low accuracy but may also remove valid 3D points belonging to the object, leading to lower accuracy in the 3D point cloud construction and affecting subsequent object size calculations.

[0005] II. Although the DBSCAN algorithm can identify noise, it is not suitable for... ε The DBSCAN algorithm is sensitive to the MinPts parameter; improper parameters can lead to missing noise points or misclassifying valid edge points as noise points. Furthermore, it requires setting different filtering thresholds for objects of different materials, resulting in low scene adaptability. Additionally, the DBSCAN algorithm has a time complexity of O(n^2). 2 The complexity cannot meet the real-time requirements of object detection in the palletizing process.

[0006] In summary, existing palletizing methods suffer from low accuracy in constructing effective 3D point clouds of objects and cannot meet real-time requirements. Summary of the Invention

[0007] The object of the present application is to provide an object stacking method, system, device and product based on visual perception, which is a full-process object stacking method and can accurately calculate the size of the object to facilitate accurate execution of the stacking operation.

[0008] To achieve the above object, the present application provides the following solutions.

[0009] In a first aspect, the present application provides an object stacking method based on visual perception, comprising:

[0010] obtaining an RGB-D image of a target object at the end of a conveying belt, the RGB-D image being obtained by taking a photograph of the target object from above by a depth camera;

[0011] generating a first three-dimensional point cloud of the target object in the coordinate system of the depth camera from the RGB-D image and converting it into a second three-dimensional point cloud in the world coordinate system;

[0012] projecting the second three-dimensional point cloud along three coordinate axes respectively to obtain three two-dimensional density heat maps;

[0013] For any one of the two-dimensional density heat maps: calculating the global density of the two-dimensional density heat map; setting a circle with a preset radius as the center of each two-dimensional point in the two-dimensional density heat map, the preset radius being 2-3 times the average point distance of the two-dimensional density heat map; calculating the local density in each circle; for any two-dimensional point, if the local density in the corresponding circle is less than the global density, the two-dimensional point is removed from the two-dimensional density heat map;

[0014] calculating the maximum two-dimensional connected region in each of the two-dimensional density heat maps respectively, retaining the three-dimensional points in the second three-dimensional point cloud that are projected into three of the maximum two-dimensional connected regions to form a target three-dimensional point cloud;

[0015] determining the size of the target object according to the target three-dimensional point cloud, and determining a target placement area of the target object in the stacking area according to the size of the target object.

[0016] In a second aspect, the present application provides an object stacking system based on visual perception, comprising:

[0017] The image acquisition module is configured to obtain an RGB-D image of a target object at the end of a conveying belt, the RGB-D image being obtained by taking a photograph of the target object from above by a depth camera;

[0018] The point cloud generation module is configured to generate a first three-dimensional point cloud of the target object in a coordinate system of the depth camera according to the RGB-D image and convert the first three-dimensional point cloud into a second three-dimensional point cloud in a world coordinate system, project the second three-dimensional point cloud along three coordinate axes respectively to obtain three two-dimensional density heat maps, for any one of the two-dimensional density heat maps, calculate a global density of the two-dimensional density heat map, set a circle with a preset radius as a center for each two-dimensional point in the two-dimensional density heat map, the preset radius being 2-3 times of an average point distance of the two-dimensional density heat map, calculate a local density in each circle, for any two-dimensional point, if the local density in the circle corresponding to the two-dimensional point is less than the global density, the two-dimensional point is removed from the two-dimensional density heat map, and calculate a maximum two-dimensional connected region in each of the two-dimensional density heat maps respectively, retain three-dimensional points in the second three-dimensional point cloud that are projected to fall in three maximum two-dimensional connected regions to form a target three-dimensional point cloud.

[0019] The feature recognition module is configured to determine a size of the target object according to the target three-dimensional point cloud.

[0020] The region determination module is configured to determine a target placement region of the target object in the stacking region according to the size of the target object.

[0021] In a third aspect, the present application provides a computer device, comprising a memory, a processor, a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the object stacking method based on visual perception according to any one of the above.

[0022] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the object stacking method based on visual perception according to any one of the above.

[0023] In a fifth aspect, the present application provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of the object stacking method based on visual perception according to any one of the above.

[0024] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0025] The application provides a visual perception-based object stacking method, system, device and product. The object stacking method provides a full-process processing step, including obtaining three-dimensional pose information of objects at the end of a conveyor belt, then accurately measuring the size of the objects, and finally dynamically calculating the optimal placement position according to the size of the objects. In the step S130 and the step S140, the three-dimensional point cloud of the target object (express) can be effectively extracted, and the noise points on the surface of the conveyor belt can be removed. The mechanism is as follows: the express is a three-dimensional entity, and the conveyor belt is a two-dimensional plane. The express point cloud can form a complete connected region in the three projection planes, while the noise points on the surface of the conveyor belt can only form a connected region in the x-y plane. In the y-z and z-x planes, the connected region is compressed into a thin line-shaped non-effective connected region. Through the screening rule of the intersection of the three-plane maximum connected regions, even if the conveyor belt has a height deviation or the conveyor belt is not completely horizontal, the background points adhered to the package can be completely filtered, and the height threshold filtering problem of height deviation background that cannot be processed is solved. In addition, the method adopts a global density + local density comparison adaptive screening rule, without manual parameter setting, the edge effective points of the target object can be reserved, the defect that the edge effective points are misjudged as noise in the traditional method is avoided, and the point cloud of various express packages such as cartons, plastics and cloth can be completely extracted. At the same time, the step S130 and the step S140 have a linear time complexity O (n): first, most of the background noise is quickly filtered through density screening, and then the pure point set is subjected to connected analysis. Compared with the O (n 2 ) complexity of the traditional DBSCAN, the calculation efficiency is improved by 1-2 orders of magnitude, and the industrial real-time measurement demand can be met. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0027] Figure 1 It is a flowchart of a visual perception-based object stacking method in an embodiment of the present application.

[0028] Figure 2 It is an example diagram of the feature recognition process of the target object in an embodiment of the present application.

[0029] Figure 3 It is an example diagram of the center of gravity stability judgment of the target object in an embodiment of the present application.

[0030] Figure 4 It is an example diagram of the target placement area selection in an embodiment of the present application.

[0031] Figure 5 This is an example diagram of an object palletizing process in one embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] In one embodiment of this application, a visual perception-based object palletizing method is provided, which can be executed by a computer device and specifically includes steps S110 to S150.

[0035] Step S110: Obtain the RGB-D image of the target object at the end of the conveyor belt. The RGB-D image is obtained by taking a top-down shot of the target object using a depth camera.

[0036] Step S120: Generate a first three-dimensional point cloud of the target object in the coordinate system of the depth camera based on the RGB-D image and convert it into a second three-dimensional point cloud in world coordinates.

[0037] In this step, in order to improve the construction efficiency of the target 3D point cloud and reduce the interference of other objects in the RGB-D image, the target object can be identified and located first using a target detection algorithm. Then, the detection box output by the target detection algorithm is cropped to obtain the region of interest containing the target object. Finally, the target 3D point cloud is generated based on the region of interest.

[0038] For example, an Intel RealSense D435i binocular camera can be used to acquire RGB-D images of the target object, and then target detection algorithms such as R-CNN series, YOLO series, and DETR can be used to identify and locate the target object.

[0039] In this embodiment, the first three-dimensional point cloud of the target object in the coordinate system of the depth camera is generated based on the RGB-D image and then transformed into a second three-dimensional point cloud in world coordinates, specifically including steps S121, S122 and S123.

[0040] Step S121 involves using an object detection algorithm to detect the RGB-D image and generate detection boxes for the target object. Based on these detection boxes, the RGB-D image is cropped to obtain a region of interest containing the target object. This step reduces the amount of point cloud data processed during size recognition and shortens the point cloud processing time.

[0041] Step S122: Generate the three-dimensional coordinates of each pixel in the region of interest based on the depth information of the RGB-D image and the intrinsic parameter matrix of the depth camera, and generate the color of each pixel in the region of interest based on the color information of the RGB-D image. Combine the three-dimensional coordinates and colors of each pixel to form the first three-dimensional point cloud in the coordinate system of the depth camera.

[0042] Specifically, after cropping the RGB-D image, each pixel in the region of interest is traversed. For each pixel, its depth value is read, and using the intrinsic parameter matrix of the depth camera, the pixel's two-dimensional coordinates are calculated. u , v Back projection into three-dimensional coordinates ( X , Y , Z This process involves reading the color of the corresponding pixel from the RGB-D image and assigning it to each 3D point. All 3D points and color information are then combined to form the first 3D point cloud in the coordinate system of the depth camera. The formula for calculating the coordinates of a 3D point is:

[0043]

[0044] in, c x , c y , f x and f y These are all intrinsic parameters of the depth camera. d This represents the depth value of a pixel.

[0045] Step S123: Convert the first 3D point cloud from the coordinate system of the depth camera to the second 3D point cloud in the world coordinate system.

[0046] The world coordinate system refers to the coordinate system used to determine the pose of the target object in the actual scene. It is also the working coordinate system of the robot arm, facilitating subsequent robot operation and global planning. Therefore, the first 3D point cloud needs to be transformed from the depth camera's coordinate system to the world coordinate system. The transformation principle is homogeneous coordinate transformation. Specifically, a 4×4 transformation matrix is ​​used to transform the first 3D point cloud from the depth camera coordinate system to the world coordinate system. The points in the depth camera coordinate system are defined as follows: The corresponding point in the world coordinate system is: The two satisfy the following relationship:

[0047]

[0048] wherein, T is a transformation matrix, R is a 3x3 rotation matrix, t is a 3x1 translation vector.

[0049] It is considered that there are still some background points in the second three-dimensional point cloud, i.e. three-dimensional points not belonging to the target object. In order to improve the accuracy of subsequent processing, these background points will also be filtered in step S130 in this embodiment.

[0050] In step S130, the second three-dimensional point cloud is projected along three coordinate axes to obtain three two-dimensional density heat maps respectively; for any one two-dimensional density heat map: the global density of the two-dimensional density heat map is calculated; a circle with a preset radius is set with each two-dimensional point in the two-dimensional density heat map as the center, and the preset radius is 2-3 times the average point distance of the two-dimensional density heat map; the local density in each circle is calculated; for any two-dimensional point, if the local density in the circle corresponding to the two-dimensional point is less than the global density, the two-dimensional point is removed from the two-dimensional density heat map.

[0051] wherein, the maximum two-dimensional connected region in each two-dimensional density heat map is calculated respectively, specifically including: for any one two-dimensional density heat map: two two-dimensional points with a Euclidean distance less than the connected judgment distance are determined to be connected, each connected region in the two-dimensional density heat map is determined based on each pair of connected two-dimensional points, and the connected region containing the most two-dimensional points is taken as the maximum two-dimensional connected region; wherein, the connected judgment distance is the average point distance of the two-dimensional density heat map.

[0052] Specifically, the projection of the second three-dimensional point cloud along the three coordinate axes is to project the second three-dimensional point cloud onto the x-y, x-z and z-y orthogonal planes. It should be noted that the three orthogonal planes are coordinate planes in the world coordinate system.

[0053] Exemplarily, the two-dimensional point removal process is described in detail as follows taking the projection onto the x-y plane as an example:

[0054] The projection obtains a two-dimensional point set (a set of two-dimensional points in the two-dimensional density heat map) For each two-dimensional point, the local density inside the circle with it as the center is calculated:

[0055]

[0056] wherein, is the local density inside the i-th circle, k i is the total number of two-dimensional points inside the i-th circle, dfor a preset radius, S d for a preset radius d of a circle, the i-th circle is a circle with the i-th two-dimensional point as the center.

[0057] The global density calculation formula of the two-dimensional density heat map is:

[0058]

[0059]

[0060] wherein, is the global density of the two-dimensional density heat map, n is the total number of two-dimensional points in the two-dimensional density heat map, which is also the total number of three-dimensional points in the second three-dimensional point cloud, S is the area of the two-dimensional density heat map, x max and x min are the maximum and minimum projection values of the second three-dimensional point cloud on the x-axis, respectively, y max and y min are the maximum and minimum projection values of the second three-dimensional point cloud on the y-axis, respectively.

[0061] When the local density of a circle is less than the global density, the two-dimensional point at the center of the circle is removed. In essence, the above is to calculate the neighborhood density of each two-dimensional point.

[0062] As described above, the two-dimensional point removal process for the two-dimensional density heat map on the x-y plane is described. The two-dimensional point removal processes for the other two planes are the same.

[0063] Through the above steps, three two-dimensional density heat maps after removing noise points can be obtained. Then, the maximum two-dimensional connected region in each two-dimensional density heat map is calculated.

[0064] For each two-dimensional density heat map, first, the connected relationship between any two two-dimensional points is calculated to determine the connected regions. Specifically, the BFS (Breadth-First Search) algorithm can be used to traverse all points in the two-dimensional density heat map to find all connected two-dimensional points, which form a connected region. For example, two-dimensional point A is connected to two-dimensional point B, two-dimensional point B is connected to two-dimensional points C and D, and two-dimensional point D is connected to two-dimensional point E, then two-dimensional points A / B / C / D / E form a connected region.

[0065] It should be noted that the calculation of the average point distance of the two-dimensional density heat map in the above step S130 is as follows: first, the distance between each two-dimensional point and its nearest two-dimensional point is calculated, and then the distance corresponding to each two-dimensional point is averaged to obtain the average point distance.

[0066] Step S140, respectively calculate the maximum two-dimensional connected region in each two-dimensional density heat map, and retain the three-dimensional points in the second three-dimensional point cloud which are projected in the three maximum two-dimensional connected regions to form a target three-dimensional point cloud.

[0067] Through steps S130 and S140 (multi-view density connectivity analysis), the 3D point cloud of the target object (express delivery) can be effectively extracted, while noise points on the conveyor belt surface are removed. The mechanism lies in the fundamental difference between the express delivery being a 3D entity and the conveyor belt being a quasi-2D plane. The express delivery point cloud can form complete connected regions in all three projection planes, while the noise points on the conveyor belt surface only form connected regions in the xy plane. In the yz and zx planes, they are compressed into thin, ineffectively connected linear regions. Specifically, in overhead shots, depth cameras rarely capture the bottom of the target object, meaning the 3D points at the bottom of the target object generally do not exist. Therefore, although the noise points on the conveyor belt surface and the projection of the target object in the xy plane can form connected regions, their projections in the yz and zx planes are essentially discontinuous; that is, the projection of the conveyor belt surface is a relatively isolated straight line. By employing a filtering rule based on the intersection of the maximum connected regions in three planes, background points adhered to packages (adhered in the vertical projection) can be thoroughly filtered out even if the conveyor belt has height deviations or is not perfectly horizontal, thus solving the problem that height threshold filtering cannot handle backgrounds with height deviations. Addressing the uneven point cloud density caused by the diverse materials used in express delivery, this method uses an adaptive filtering rule combining global density and local density comparison. This retains valid edge points of the target object without requiring manual parameter settings. Edge points are categorized as valid and invalid. The density of valid edge points is slightly lower than, but close to, the point density of the main body of the target object. Invalid edge points are distorted and spread outwards; introducing invalid edge points increases the point cloud size, leading to an increase in the calculated object size. The density of invalid edge points and noise points is significantly lower than that of valid edge points and the main body of the target object. When calculating the global density, the presence of invalid and noise points results in a global density between that of invalid and valid edge points, effectively eliminating them. Since global density is used as the filtering threshold, the above filtering criteria are applicable to target objects of any material. In this algorithm, the global density of target objects made of different materials varies, and all fall between the effective and ineffective edge point densities of the target object. Traditional DBSCAN algorithms require manual setting of a filtering threshold, making the filtering accuracy heavily dependent on this threshold setting. If the threshold is set appropriately (e.g., precisely between the effective and ineffective edge point densities), the same technical effect as in this embodiment can be achieved. However, an inappropriate threshold setting severely impacts the point cloud filtering accuracy. For example, if the filtering threshold is set too low, noise points and ineffective edge points will not be sufficiently removed; if the threshold is set too high, effective edge points will be removed, and even some 3D points of the main body of the target object will be removed, resulting in the calculated object size being smaller than the actual value. Therefore, the accuracy of traditional DBSCAN algorithms is highly dependent on the setting of the filtering threshold.More importantly, the point density of the main part and the effective edge point density of the target object of different materials are different, such as the point density of the foam material is obviously lower than that of the wooden box. Therefore, a fixed screening threshold cannot be applied to target objects of different materials. For example, a higher screening threshold can effectively eliminate the noise points and edge invalid points of the wooden box, but it may also eliminate the edge effective points of the foam package and even the three-dimensional points of the main part. Correspondingly, a lower screening threshold can effectively eliminate the edge invalid points and noise points of the foam package, but it may not be able to eliminate the edge invalid points of the wooden box (the overall point density is higher, and the edge invalid point density is also higher). In the palletizing scene, the target objects on the conveyor belt are not of uniform material, and the next one may be a wooden box after a foam package. As a real-time recognition scene, it is difficult to change the screening threshold according to the current target object without interruption in real time, and it also increases the labor cost and increases the error rate.

[0068] At the same time, steps S130 and S140 are both linear time complexity O(n): first, filter most of the background noise through density screening, and then perform connected analysis on the pure point set. Compared with the O(n 2 ) complexity of the traditional DBSCAN, the calculation efficiency is improved by 1-2 orders of magnitude, which can meet the real-time measurement requirements of industry.

[0069] Through the above steps, the three-dimensional point cloud of the output target object (target three-dimensional point cloud) can be obtained more accurately.

[0070] Step S150, according to the target three-dimensional point cloud, determine the size of the target object, and according to the size of the target object, determine the target placement area of the target object in the palletizing area.

[0071] Among them, according to the target three-dimensional point cloud, the size of the target object is determined specifically as follows:

[0072] Determine the optimal bounding box of the target three-dimensional point cloud, take the length and width of the optimal bounding box as the length and width of the target object respectively, take the difference between the maximum depth value of the target three-dimensional point cloud and the height value of the conveyor belt as the height of the target object, and determine the grabbing point of the target object according to the upper surface center point of the optimal bounding box. The optimal bounding box is the smallest bounding box of the target three-dimensional point cloud, and the target object is grabbed by the mechanical hand to the palletizing area through the grabbing point.

[0073] In this step, different bounding boxes of the target three-dimensional point cloud can be determined by a grid search algorithm, and the one with the smallest volume is taken as the optimal bounding box. The principle is that the bounding box with the smallest volume should be the one that best fits the target three-dimensional point cloud, and its size and shape are closest to the target three-dimensional point cloud. Since the RGB-D image is a top-down image, the length and width of the target three-dimensional point cloud are the length and width of the target object, so the length and width of the optimal bounding box can be taken as the length and width of the target object. In the height direction, the depth camera cannot capture the bottom and interior of the target object, and the target three-dimensional point cloud is only the point cloud of the visible part of the target object from the top view (basically the top point cloud of the target object), so the height of the target three-dimensional point cloud is different from the height of the target object. Considering that the target object is placed on the conveyor belt, the difference between the minimum depth value of the target three-dimensional point cloud (corresponding to the highest point in the target object) and the depth value of the conveyor belt is taken as the height of the target object.

[0074] In this embodiment, the optimal bounding box of the target three-dimensional point cloud is determined, which specifically includes steps S151, S152 and S153.

[0075] Step S151: Randomly selecting three three-dimensional points in the target three-dimensional point cloud to form a plurality of candidate planes; for each candidate plane, calculating the comprehensive score of all three-dimensional points in the target three-dimensional point cloud relative to the candidate plane:

[0076]

[0077] wherein, S i is the comprehensive score of the i-th three-dimensional point relative to the candidate plane, , and are three weights, and the sum of the three weights is 1, d i and are the Euclidean distance from the i-th three-dimensional point to the candidate plane and its standard deviation, respectively, and are the normal vectors of the candidate plane and the i-th three-dimensional point, respectively, is the field point density of the i-th three-dimensional point, is the average point density of the candidate plane.

[0078] Preferably, , and are 0.5, 0.4 and 0.1, respectively. Exemplarily, for the three-dimensional point the distance from the plane distance is:

[0079]

[0080] Step S152, determine the three-dimensional points with a comprehensive score greater than the score threshold relative to the candidate plane as inliers of the candidate plane, and determine the candidate plane with the most inliers as the target plane.

[0081] Step S151 and step S152 are to determine an optimal representative plane of the main surface of the target object, and then calculate the covariance matrix of the target three-dimensional point cloud based on all points on the plane.

[0082] It should be noted that the traditional RANSAC (RANdom SAmple Consensus) plane fitting method only relies on the distance of the point to the plane as the basis for determining the inliers, and has low discrimination for noise points and edge points, and random sampling is easy to draw noise points, resulting in insufficient fitting accuracy. The embodiment proposes a robust RANSAC plane fitting method with multiple feature constraints (corresponding to steps S151 and S152) for the object point cloud volume measurement scene, effectively solving the above problems.

[0083] In the express delivery scene, the low-density noise points left on the conveyor belt and the effective sparse points on the package edge may be highly similar in geometric features. For example, when the residual points fall near the package edge, the angle between their normal vector and the normal vector of the package plane may satisfy the threshold, and the distance to the plane may also meet the standard. The existing normal vector + distance scheme only relies on geometric features and cannot distinguish between these two types of points, which may misjudge this type of noise points as effective inliers, causing the fitted plane to be biased by background points, and ultimately affecting the accuracy of package size measurement. However, with the addition of the density feature, this problem is fundamentally solved: the noise points left on the conveyor belt are essentially background interference, and their local density is much lower than the stable density of the main points and effective edge points of the package. In the comprehensive score calculation, the density weight of this type of point will tend to 0, so even if the geometric feature meets the standard, the comprehensive score will be lower than the threshold, and thus it will be accurately excluded. This design is not simply adding features, but rather it provides a feasible basis for distinguishing density for the core pain point of the express delivery scene, which is that the geometric features of the background and effective edge points are highly similar, and it completely solves the technical problem that the existing scheme cannot distinguish between three-dimensional points with similar geometric features but different attributes.

[0084] Step S153, calculate the covariance matrix of the target three-dimensional point cloud based on all three-dimensional points on the target plane, and determine the optimal bounding box of the target three-dimensional point cloud based on the covariance matrix.

[0085] Covariance matrix C The calculation formula is:

[0086]

[0087] in, p i Let i be the i-th 3D point in the target plane. N The total number of three-dimensional points in the target plane. The center point is located in the target plane.

[0088] Then, the covariance matrix is ​​decomposed into eigenvalues. .in, v j For the first j The directions of the three largest eigenvalues ​​are used as the three axes to construct the local coordinate system of the target 3D point cloud. Essentially, the three axes of the local coordinate system correspond to the length, width, and height directions of the bounding box of the target 3D point cloud.

[0089] Furthermore, the optimal bounding box of the target 3D point cloud is determined based on the covariance matrix, specifically including: performing eigenvalue decomposition on the covariance matrix to obtain a local coordinate system containing three coordinate axes for the target 3D point cloud; rotating the local coordinate system around the Z-axis of the local coordinate system by a preset step size of 360°; and at each rotation angle, determining the bounding box and calculating its volume based on the difference between the maximum and minimum values ​​of the projections of the target 3D point cloud onto the three coordinate axes, where the three differences are the length, width, and height of the bounding box, respectively.

[0090] For example, the preset step size is 1°. In this step, a grid search strategy is used to perform a 360-degree rotation search around the Z-axis with a step size of 1° to find the optimal rotation angle that minimizes the bounding box volume. The search process is implemented as follows:

[0091] Construct the rotation matrix. The rotation matrix around the Z-axis is defined as follows:

[0092]

[0093] in, It is a rotation angle around the Z-axis of . The rotation matrix, Let be the rotation angle about the Z-axis. , , and These are the cosine and sine values ​​of the rotation angle, respectively.

[0094] Initialize the local coordinate system ,make , as well as . v 1. v 2 and v 3 represents three coordinate axes, corresponding to the length, width, and height directions of the bounding box, respectively, which are the length axis, width axis, and height axis.

[0095] Grid search optimization: for each rotation angle , the rotated axis vector calculation is performed:

[0096]

[0097] wherein, is the local coordinate system after the rotation angle.

[0098] The projection calculation of the target three-dimensional point cloud on the three coordinate axes respectively:

[0099]

[0100] wherein, is the coordinate matrix of the target three-dimensional point cloud after projection to the coordinate system , is the coordinate matrix of the target three-dimensional point cloud itself, P is the center point of the target three-dimensional point cloud.

[0101] Through the above projection calculation, the projection coordinates of the target three-dimensional point cloud on the three coordinate axes respectively can be obtained, so as to determine the maximum and minimum projection values on each axis. For each coordinate axis, the difference between the maximum and minimum projection values is the size of the bounding box in the direction of the coordinate axis, and finally the sizes of the bounding box in three directions can be determined, and the product of the three sizes is the volume of the bounding box.

[0102] By traversing each rotation angle, the volume of the bounding box under each rotation angle can be determined, and the optimal rotation angle that makes the volume of the bounding box minimum is determined.

[0103]

[0104] wherein, is the volume of the bounding box under the rotation angle, and the above formula indicates that the rotation angle that makes the volume of the bounding box minimum is taken as the optimal rotation angle.

[0105] The bounding box under the optimal rotation angle is the optimal bounding box, and the optimal local coordinate system is determined based on the optimal rotation angle. The three axis directions of the optimal local coordinate system are the length, width and height directions of the optimal bounding box. The optimal local coordinate system can be used to determine the optimal bounding box and the pose of the target object.

[0106] Then the bounding box with the minimum volume is determined as the optimal bounding box. The size calculation of the optimal bounding box is as follows:

[0107]

[0108] wherein,​ The projections of the target three-dimensional point cloud on the three coordinate axes of the optimal local coordinate system are determined, and the maximum and minimum values of the projections on the three coordinate axes are calculated respectively by the following functions:

[0109]

[0110]

[0111] wherein, and represent the minimum and maximum values of the projections on the coordinate axes , respectively. . and represent the minimum and maximum values of the projections on the coordinate axes , respectively, within the range .

[0112] Finally, the length and width of the optimal bounding box are taken as the length and width of the target object, respectively:

[0113]

[0114]

[0115] and the difference between the minimum depth value of the target three-dimensional point cloud and the depth value of the transmission belt is taken as the height of the target object. It should be noted that the height of the target object can be calculated directly from the corresponding depth value in the depth camera, or the maximum value of the projection of the target three-dimensional point cloud on the height axis minus the height value of the transmission belt:

[0116]

[0117] wherein, , and are the length, width and height of the optimal bounding box, a is the height value of the transmission belt. In an example scenario, 0.2 is taken. and represent the maximum and minimum values of the projections of the target three-dimensional point cloud on the length axis, and represent the maximum and minimum values of the projections of the target three-dimensional point cloud on the width axis, represents the maximum value of the projection of the target three-dimensional point cloud on the height axis.

[0118] After the optimal rotation angle is determined, the pose of the optimal bounding box is the pose of the target object, and the angle of the longest axis v 1 is the orientation angle :

[0119]

[0120] wherein, and are the element in the 0th row and 0th column and the element in the 0th row and 0th column of respectively. denotes the four-quadrant arctangent function. It is to be noted that, the row and column indices in

[0121] Generally, the grasping position of the robot arm is the center point of the upper surface of the target object. Since the center point of the upper surface of the optimal bounding box corresponds to the center point of the upper surface of the target object, the grasping point of the target object is determined according to the center point of the upper surface of the optimal bounding box. Specifically, the position point of the center point of the upper surface of the optimal bounding box in the world coordinate system is determined as the grasping point of the target object.

[0122] First, the coordinates of the center point of the upper surface of the optimal bounding box in the optimal local coordinate system need to be calculated c top :

[0123]

[0124] Then, the coordinates of the center point of the upper surface of the optimal bounding box are converted from the optimal local coordinate system to the world coordinate system.

[0125]

[0126] wherein, p grasp denotes the coordinates of the grasping center point in the world coordinate system.

[0127] Through the above steps, based on the optimal bounding box of the target object, the size, grasping point and pose of the target object can be determined. In the subsequent grasping process, the robot arm grasps the target object through the grasping point, and can adjust the pose of the target object according to the placement position during the placement process.

[0128] The above steps S120 and S150 are the feature recognition process of the target object. Referring to Figure 2 , an example of the feature recognition process of the target object is shown in Figure 2 . First, the RGB image and the depth image (RGB-D image) of the express (target object) are cropped and the point cloud is extracted according to the detection frame, and then the multi-view density connected analysis is performed on the point cloud to remove the noise point cloud (the set of background points in the point cloud). Secondly, the grid search algorithm based on the oriented bounding box is used to find the smallest bounding box, and finally the size, pose and grasping point of the express are obtained by analyzing the features of the bounding box.

[0129] In the step of determining the target placement region of the target object in the stacking region according to the size of the target object: first, based on the stability judgment standard, find out the candidate placement region that meets the stable placement, and then determine the target placement region in the candidate placement region. Specifically, it includes steps S154 and S155.

[0130] In step S154, based on the height map of the stacking region, the contact surface between the current bottom surface and the stacking region when the target object is placed in different sub-regions of the stacking region is determined in the first and second pose states of the target object, and the sub-region in which the center of gravity of the target object falls on the contact surface and the contact surface is greater than half of the current bottom surface is taken as the candidate placement region. The size of the sub-region is the same as that of the current bottom surface, and the first and second pose states are two pose states formed by interchanging the length and width of the bottom surface of the target object.

[0131] For the height map of the stacking region, the RGB-D image of the stacking region can be cropped to obtain the region of interest, and each pixel point in the region of interest is traversed. For each pixel coordinate (x, y), read its depth value depth, and project the two-dimensional pixel coordinate to the three-dimensional space according to the depth camera intrinsic matrix to obtain the three-dimensional point coordinate (X, Y, Z). At the same time, the color information of the corresponding pixel is obtained from the RGB-D image, and the three-dimensional point is assigned. Integrate all three-dimensional points and their color information to generate a complete point cloud of the stacking region. Voxelize the point cloud and discretize it into a three-dimensional grid of length x width x height, and calculate the point cloud height value in each grid cell. Then, project the three-dimensional grid along the height direction to generate a two-dimensional height table, i.e. the height map of the stacking region. Each position in the two-dimensional height table stores the height value of the corresponding grid cell, and the height value of each grid cell is the height value of the corresponding position in the stacking region.

[0132] After determining the size of the target object and the height map of the stacking region, the placement position of the target object is calculated. The selection of the placement position not only ensures the feasibility of the object placement, but also ensures the space utilization of the object. Therefore, the object center of gravity and bottom surface contact area joint determination method is used to ensure the stability of the object in this embodiment.

[0133] Object stability determination is mainly used in automated stacking, handling and other scenarios, and the purpose is to ensure that the placed object can maintain balance at the target position and will not fall due to insufficient support or center of gravity offset. The core idea is that the bottom surface of the object must have sufficient support area, and the center of gravity must fall within the support area. When selecting the placement position of the object, first, the actual contact area between the bottom surface of the object and the lower support surface is counted.

[0134] The contact surface of the current bottom surface with the stacking area when the target object is placed in different sub-areas in the stacking area is determined, specifically comprising: first determining a sub-area in the stacking area which is the same as the bottom surface of the target object, then determining the height of each position point in the sub-area based on the height map of the stacking area, determining the highest position points in the sub-area, and all the highest position points constitute the support area, i.e. the contact surface of the stacking area and the current bottom surface.

[0135] After obtaining the contact surface, it is judged whether the geometric center of the bottom surface of the target object falls within the convex hull or boundary of the support area. If yes, it is considered that the center of gravity of the target object falls on the contact surface. If no, it is considered that the center of gravity of the target object does not fall on the contact surface. At this time, the object may be tilted and is not allowed to be placed. Figure 3 An example is provided for the contact surface and whether the center of gravity of the target object falls within the contact surface.

[0136] Exemplarily, assuming that the geometric center coordinates of the bottom surface of the target object are , the coordinates of each position point on the contact surface are , and the judgment condition is:

[0137]

[0138]

[0139] wherein, x min and x max respectively x i are the minimum and maximum values in y min and y max respectively y i are the minimum and maximum values in

[0140] If the area of the contact surface is less than half of the current bottom surface at the same time, it is considered that the support is insufficient and placement is not allowed. Only the sub-area that meets all the above conditions at the same time is considered to be a stable and feasible placement area.

[0141] Through the above steps, all feasible placement areas corresponding to the target object in different pose states (two pose states formed by interchanging the length and width of the bottom surface, which are realized by rotating the target object by 90 degrees around the height axis) can be obtained, and these feasible placement areas constitute a set of candidate placement areas.

[0142] In step S155, the candidate placement area with the lowest height is taken as the first candidate placement area, and the target placement area is determined in the first candidate placement area.

[0143] In this step, the target object is preferentially placed in a lower position to ensure stability. Therefore, the candidate placement region with the lowest height is selected as the first candidate placement region. If there is only one first candidate placement region, this region is selected as the target placement region. If there are multiple first candidate placement regions, one is selected from the first candidate placement regions as the target placement region. Further, in the horizontal direction, the priority of the X-axis and Y-axis in the world coordinate system can be adjusted according to the actual scene. For example, when the priority of the X-axis is higher than that of the Y-axis, the candidate placement region with the minimum X-coordinate is preferentially selected from the first candidate placement regions as the second candidate placement region. If there is only one second candidate placement region, this region is selected as the target placement region. If there are multiple second candidate placement regions, the second candidate placement region with the minimum Y-coordinate is selected from the multiple second candidate placement regions as the target placement region.

[0144] Referring to Figure 4 which provides an example of selecting a target placement region (T) from candidate placement regions (C) Figure 4 wherein the candidate positions represent vertices of the candidate placement regions (C) Figure 4 and the target position represents a vertex of the target placement region (T).

[0145] It should be noted that after the target placement region is determined, the robot needs to adjust the pose of the target object so that the corresponding surface plane serves as the bottom surface to place the target object on the target placement region. The bottom surface of the target object needs to be completely matched with the target placement region.

[0146] As above, a specific embodiment of the object stacking method based on visual perception is fully described. The whole process closed-loop control from object recognition, size estimation, angle estimation, grasp point estimation to stability determination, position planning is realized, solving the problem of fragmentation of each link of the traditional stacking system. Among them, the specific recognition steps of the object size are also specifically given: respectively project the second three-dimensional point cloud along the three coordinate axes to obtain three two-dimensional density heat maps; for any one two-dimensional density heat map: calculate the global density of the two-dimensional density heat map; set a circle with a preset radius as the center of each two-dimensional point in the two-dimensional density heat map, and the preset radius is 2-3 times the average point distance of the two-dimensional density heat map; calculate the local density in each circle; for any two-dimensional point, if the local density in the corresponding circle is less than the global density, the two-dimensional point is removed from the two-dimensional density heat map; calculate the maximum two-dimensional connected region in each two-dimensional density heat map, respectively, and retain the three-dimensional points in the second three-dimensional point cloud that are projected simultaneously in the three maximum two-dimensional connected regions to form a target three-dimensional point cloud, which can efficiently and accurately determine the size of the object. And the double-criteria stability determination steps (contact area greater than half of the object bottom surface and object center of gravity in the contact surface) are specifically given, which greatly improves the stacking stability. The heuristic placement strategy based on height map (successively preferentially selecting the placement area with the smallest height, the smallest X coordinate and the smallest Y coordinate) has strong generalization ability and can handle objects of different sizes, weights and packaging materials, ensuring that the space utilization rate is more than 73%, which is better than the conventional heuristic algorithm.

[0147] Based on the object stacking method, the application exemplarily provides an object stacking process. Referring to Figure 5 , first of all, the end sensor of the conveyor belt is detected, and whether there is an object at the end of the conveyor belt is determined based on the signal of the sensor. If there is, on the one hand, the object category is identified by a detection model (target detection algorithm) and a detection box is drawn, different categories of objects are stacked using different types of trays, and then the minimum bounding box (optimal bounding box) is found based on the bounding box optimization algorithm of grid search, the features of the bounding box are analyzed, and the express posture, express size and grasping position (grasp point) are calculated. On the other hand, the height map of the stacking area is determined according to the point cloud of the stacking area. Combined with the height map of the stacking area and the express size, the height map of the stacking area is traversed, the center of gravity is determined and the contact area is determined for each position (sub-region), so as to determine whether the express is stable when placed in the position. If it is stable, the position is put into the candidate position set, and the target position is obtained based on the heuristic placement strategy of the height map and the express is placed. Finally, it is judged whether there is an object to be stacked, and if there is, the above process is executed in a loop until it ends.

[0148] In an exemplary embodiment, an object stacking system based on visual perception is also provided, comprising an image acquisition module, a point cloud generation module, a feature recognition module and a region determination module.

[0149] The image acquisition module is configured to acquire an RGB-D image of the target object at the end of the conveying belt, the RGB-D image being obtained by shooting the target object from above by a depth camera.

[0150] The point cloud generation module is configured to generate a first three-dimensional point cloud of the target object in a coordinate system of the depth camera from the RGB-D image and convert the first three-dimensional point cloud into a second three-dimensional point cloud in a world coordinate system; project the second three-dimensional point cloud along three coordinate axes respectively to obtain three two-dimensional density heat maps; for any one of the two-dimensional density heat maps: calculate a global density of the two-dimensional density heat map; set a circle with a preset radius around each two-dimensional point in the two-dimensional density heat map, the preset radius being 2-3 times of an average point distance of the two-dimensional density heat map; calculate a local density in each circle; for any two-dimensional point, if the local density in the circle corresponding to the two-dimensional point is less than the global density, the two-dimensional point is removed from the two-dimensional density heat map; calculate a maximum two-dimensional connected region in each of the two-dimensional density heat maps respectively, and retain three-dimensional points in the second three-dimensional point cloud that are projected to fall in three maximum two-dimensional connected regions to form a target three-dimensional point cloud.

[0151] The feature recognition module is configured to determine a size of the target object according to the target three-dimensional point cloud.

[0152] The region determination module is configured to determine a target placement region of the target object in the stacking region according to the size of the target object.

[0153] It should be noted that each module in the object stacking system based on visual perception is respectively configured to implement each step of the object stacking method based on visual perception in the above embodiments, and thus the specific description of the object stacking system can refer to the above object stacking method.

[0154] In an exemplary embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method embodiments when executing the computer program.

[0155] In an exemplary embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0156] In an exemplary embodiment, a computer program product is provided, including a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0157] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0158] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Among them, any reference to memory, database or other medium used in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

[0159] The database involved in each embodiment provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in each embodiment provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0160] Each technical feature of the above embodiments can be combined arbitrarily. In order to make the description simple, not all possible combinations of each technical feature in the above-mentioned embodiments are described, however, as long as the combination of these technical features does not exist contradictory, it should be considered as the scope of the present application.

[0161] The principles and implementations of the present application are described in the specific examples herein, and the above examples are only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A visual perception-based object palletizing method, characterized in that, include: Acquire an RGB-D image of the target object at the end of the conveyor belt, wherein the RGB-D image is obtained by taking a top-down view of the target object using a depth camera; Generate a first three-dimensional point cloud of the target object in the coordinate system of the depth camera based on the RGB-D image and transform it into a second three-dimensional point cloud in world coordinates; Three two-dimensional density heatmaps are obtained by projecting the second three-dimensional point cloud along the three coordinate axes respectively. For any two-dimensional density heatmap: the global density of the two-dimensional density heatmap is calculated; circles with a preset radius centered on each two-dimensional point in the two-dimensional density heatmap are set, and the preset radius is 2-3 times the average point spacing of the two-dimensional density heatmap; the local density within each circle is calculated; for any two-dimensional point, if the local density within its corresponding circle is less than the global density, the two-dimensional point is removed from the two-dimensional density heatmap. Calculate the largest two-dimensional connected region in each of the two-dimensional density heatmaps, and retain the three-dimensional points in the second three-dimensional point cloud whose projections simultaneously fall into the three largest two-dimensional connected regions to form the target three-dimensional point cloud; The dimensions of the target object are determined based on the target 3D point cloud: the optimal bounding box of the target 3D point cloud is determined, and the length and width of the optimal bounding box are used as the length and width of the target object, respectively. The difference between the minimum depth value of the target 3D point cloud and the depth value of the conveyor belt is used as the height of the target object. The gripping point of the target object is determined based on the center point of the upper surface of the optimal bounding box, which is the bounding box with the smallest volume of the target 3D point cloud. The target object is gripped by the robot arm and placed in the palletizing area through the gripping point. The target placement area of ​​the target object in the palletizing area is determined based on the dimensions of the target object. Determining the optimal bounding box of the target 3D point cloud specifically includes: Multiple candidate planes are formed by randomly selecting three 3D points from the target 3D point cloud; for each candidate plane, a comprehensive score of all 3D points in the target 3D point cloud relative to the candidate plane is calculated: in, S i The comprehensive score of the i-th 3D point relative to the candidate plane is given by ω1, ω2, and ω3, which are three weights, and the sum of the three weights is 1. d i and σ d Let be the Euclidean distance from the i-th 3D point to the candidate plane and its standard deviation, respectively. and Let be the normal vectors of the candidate plane and the i-th 3D point, respectively. ρ i Let the density of the neighborhood points of the i-th 3D point be . The average point density of the candidate plane; Three-dimensional points whose comprehensive score relative to the candidate plane is greater than the score threshold are determined as interior points of the candidate plane, and the candidate plane with the most interior points is determined as the target plane; The covariance matrix of the target 3D point cloud is calculated based on all 3D points on the target plane, and the optimal bounding box of the target 3D point cloud is determined based on the covariance matrix.

2. The object palletizing method based on visual perception according to claim 1, characterized in that, Calculate the maximum two-dimensional connected region for each of the two-dimensional density heatmaps, specifically including: For any given two-dimensional density heatmap: two two-dimensional points whose Euclidean distance is less than the connectivity determination distance are considered connected; based on each pair of connected two-dimensional points, each connected region in the two-dimensional density heatmap is determined; and the connected region containing the most two-dimensional points is taken as the maximum two-dimensional connected region; wherein, the connectivity determination distance is the average point spacing of the two-dimensional density heatmap.

3. The object palletizing method based on visual perception according to claim 1, characterized in that, Determining the optimal bounding box of the target 3D point cloud based on the covariance matrix specifically includes: The covariance matrix is ​​decomposed into eigenvalues ​​to obtain a local coordinate system containing three coordinate axes for the target 3D point cloud. The local coordinate system is rotated 360° around the Z-axis of the local coordinate system with a preset step size. At each rotation angle, the bounding box is determined and its volume is calculated based on the difference between the maximum and minimum values ​​of the projection of the target 3D point cloud onto the three coordinate axes, respectively. The three differences are the length, width and height of the bounding box, respectively. The bounding box with the smallest volume is determined as the optimal bounding box.

4. The object palletizing method based on visual perception according to claim 3, characterized in that, The preset step size is 1°.

5. The object palletizing method based on visual perception according to claim 1, characterized in that, Determining the target placement area of ​​the target object in the palletizing area based on the size of the target object specifically includes: Based on the height map of the palletizing area, in the first and second pose states of the target object, the contact surface between the bottom surface of the target object and the palletizing area is determined when the target object is placed in different sub-regions of the palletizing area. The sub-regions in which the center of gravity of the target object falls on the contact surface and the contact surface is greater than half of the bottom surface of the target object are selected as candidate placement areas. The sub-regions have the same bottom surface dimensions as the target object, and the first and second pose states are two pose states formed by interchanging the length and width of the bottom surface of the target object. The candidate placement area with the lowest height is selected as the first candidate placement area, and the target placement area is determined within the first candidate placement area.

6. A visual perception-based object palletizing system, which employs the visual perception-based object palletizing method according to any one of claims 1 to 5, characterized in that, The object palletizing system includes: The image acquisition module is used to: acquire an RGB-D image of the target object at the end of the conveyor belt, wherein the RGB-D image is obtained by taking a top-down shot of the target object using a depth camera; The point cloud generation module is used to: generate a first three-dimensional point cloud of the target object in the coordinate system of the depth camera based on the RGB-D image and convert it into a second three-dimensional point cloud in world coordinates; project the second three-dimensional point cloud along the three coordinate axes to obtain three two-dimensional density heatmaps; for any two-dimensional density heatmap: calculate the global density of the two-dimensional density heatmap; set circles with a preset radius centered on each two-dimensional point in the two-dimensional density heatmap, the preset radius being 2-3 times the average point spacing of the two-dimensional density heatmap; calculate the local density within each circle; for any two-dimensional point, if the local density within its corresponding circle is less than the global density, then remove the two-dimensional point from the two-dimensional density heatmap; calculate the maximum two-dimensional connected region in each two-dimensional density heatmap, and retain the three-dimensional points in the second three-dimensional point cloud whose projections simultaneously fall within the three maximum two-dimensional connected regions to form the target three-dimensional point cloud; The feature recognition module is used to: determine the size of the target object based on the target 3D point cloud; The area determination module is used to: determine the target placement area of ​​the target object in the palletizing area based on the size of the target object.

7. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the visual perception-based object palletizing method according to any one of claims 1-5.

8. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the visual perception-based object palletizing method according to any one of claims 1-5.

Images