A constraint oriented bounding volume picking method and system for cylindrical goods

Abstract

A kind of constraint directional bounding box oriented to columnar commodity is provided, and a method and system for picking up, target commodity point cloud is converted to robot base coordinate system, and initial directional bounding box is solved;Select near vertical principal axis and force alignment world vertical axis, remove vertical component from the remaining axis, orthogonal normalization and keep right-hand coordinate system;Vertical axis is fixed and unchanged, exchange horizontal axis to make the longest side of bounding box align local horizontal first axis, and construct constraint directional bounding box;According to the selection of robot working partition, RPY posture is selected, and end pose is generated by superimposing picking offset to execute approach-picking-lifting action;When segmentation or planning fails, waist joint is disturbed to retry and set retry upper limit, the present application does not need training sample, relies on geometric prior to suppress posture deviation, and operation robustness is high, suitable for supermarket warehousing sorting robot.

Description

Technical Field

[0001] This invention relates to the field of robot vision grasping technology, and in particular to a constraint-oriented bounding box grasping method and system for cylindrical goods. Background Technology

[0002] In automated operations of retail warehousing and supermarket shelves, robots need to perform top-down grasping operations on cylindrical products such as bottles and cans placed on shelves. The robot relies on depth vision sensors to acquire images of the shelf scene, obtains point cloud data of the target cylindrical products through instance segmentation, constructs an Oriented Bounding Box (OBB) based on the point cloud, estimates the object's pose, generates the grasping pose of the robotic arm's end effector, and finally drives the robotic arm to complete the grasping action.

[0003] Oriented bounding boxes (OBB) differ from axis-aligned bounding boxes (AABB). AABB bounding boxes have their coordinate axes strictly aligned with the world coordinate system, and the bounding box follows the world coordinate system. OBB bounding boxes rotate with the object's own pose, and the three principal axes of the bounding box conform to the object's shape. For point clouds, the standard OBB solution process is as follows: calculate the covariance matrix of the point cloud; perform eigenvalue decomposition on the covariance matrix to obtain three eigenvectors and eigenvalues; the three eigenvectors are the three principal axis directions of the OBB, forming the rotation matrix R; the magnitudes of the eigenvalues ​​correspond to the dimensions of each axis of the bounding box; project all points onto the three principal axis directions, take the maximum and minimum projection values, and calculate the center of the bounding box and the extent dimensions of each axis.

[0004] The classic algorithm assumes that the point cloud is complete, has low noise, and can adequately represent the shape of the object. When the point cloud contains noise, local missing data, or a small number of outliers, the covariance matrix is ​​disturbed, and the eigenvectors (principal axis direction) are deflected. In the scenario of segmenting columnar bottles and cans on a shelf, the segmented target point cloud is often incomplete. For example, if the back of a bottle is pressed against the shelf back panel, the camera cannot capture the back point cloud; adjacent products may obstruct the view of the bottle, causing missing side point clouds; reflective metal cans may cause the depth sensor to output invalid depth, resulting in holes; and instance segmentation algorithms may occasionally missegment shelf edge pixels into the target point cloud, introducing background noise. All of these real-world factors cause the original OBB principal axis to tilt.

[0005] For gripping cylindrical items, the operating logic is to grab them from top to bottom. The item stands upright on the shelf, with its axis approximately parallel to the vertical Z-axis. Due to noise interference, the original OBB (Optical Body Tool) deflects the vertical axis, outputting an incorrect object posture, which is then transmitted to the robotic arm, causing the gripper to tilt. As the gripper tilts and falls, its two fingers are no longer parallel to the bottle's sidewall, reducing the gripping contact area and shifting the gripping force direction. This greatly increases the risk of the item colliding with the shelf, knocking over the bottle, or slipping and falling.

[0006] As for existing point cloud grasping pose-related technologies, CN122289380A discloses a point cloud grasping pose prediction method, device, computer equipment, medium, and product. This technology acquires the point cloud grasping requirements of the target environment, generates point cloud description features, obtains the coordinate information of each point cloud sampling point, inputs the point cloud description features and point cloud coordinates into a pre-trained grasping pose prediction model, and outputs the point cloud grasping pose corresponding to each point cloud sampling point. This solution is highly dependent on the quality of the training dataset and the generalization ability of the model. For specific scenarios such as shelf column bottles and cans, it does not design a constraint-oriented bounding box geometry construction process that forces vertical axis alignment and horizontal longest side alignment for segmented point clouds. When the point cloud contains noise or local missing parts, the predicted grasping pose is prone to deviation, resulting in poor grasping stability.

[0007] For example, Chinese patent CN122439185A discloses a technique for discrete rotation based on Pareto solids for oriented bounding boxes. This technique constructs oriented bounding boxes with orientations defined by Pareto solids, used for ray intersection testing of the bounding volume hierarchy (BVH) to complete BVH traversal. This technique is geared towards accelerating graphics rendering and is not applicable to robotic grasping of cylindrical goods. It cannot solve the technical problem of stabilizing the grasping posture of cylindrical goods and cannot be directly transferred to robotic shelf grasping scenarios.

[0008] Existing unconstrained oriented bounding box (OBB) calculation methods directly solve the covariance matrix of the noisy segmented point cloud to obtain the OBB principal axis. Point cloud noise and local incompleteness can cause random tilting and deflection of the OBB principal axis. For top-down grasping operations, columnar goods need to be in an approximately vertical posture. Principal axis tilt will directly cause jitter in the grasping posture of the robotic arm end effector, increasing the grasping failure rate. Furthermore, the solution of using a deep learning model to predict the grasping pose requires collecting a large number of scene samples to complete the model training for the massive number of columnar SKUs of different sizes on the shelf, resulting in high deployment and debugging costs. It does not fully utilize the geometric prior knowledge that columnar goods are placed vertically, and the model has limited generalization effect in new product and new shelf environments.

[0009] However, existing technologies urgently need a geometrically constrained OBB construction scheme for the scenario of gripping columnar goods on shelves, which can stably output the gripping pose even when there is noise in the segmented point cloud. Summary of the Invention

[0010] In view of the technical problems existing in the prior art, the present invention aims to provide a constraint-oriented bounding box grasping method and system for columnar commodities, which can stably estimate the object posture and end-grabbing pose adapted to top-down grasping operation under the condition that there is noise in the segmentation point cloud of columnar retail commodities.

[0011] According to the present invention, by constructing a constrained bounding box for the segmented point cloud, forcing the near vertical axis to align with the world vertical axis, orthogonalizing and maintaining the right-handed coordinate system, aligning the longest side with the local horizontal first axis, generating the end pose of the gripper based on the constrained OBB, and driving the robotic arm to execute a complete gripping action sequence, it is possible to suppress the posture jitter caused by point cloud noise, reduce deployment costs, and improve the success rate of gripping columnar goods from top to bottom on shelves.

[0012] Specifically, according to a first aspect of the present invention, a method for gripping a constrained, oriented bounding box for columnar goods is provided, comprising the following steps: S1. Obtain the target point cloud: Obtain the visual instance mask of the target columnar product, combine it with the depth information to generate the target point cloud, transform the target point cloud to the coordinate system of the robotic arm base, and obtain the point set P; S2. Calculate the initial orientation bounding box: Solve for the initial orientation bounding box of the point set P to obtain the initial rotation matrix R, the bounding box center c, and the size vector e; the three columns of the initial rotation matrix R correspond to the three principal axis directions of the initial OBB. S3. Select the nearest vertical axis and force alignment with the world vertical direction; set the world vertical unit vector. ; Calculate the principal axis vector of each column of the initial rotation matrix R. Vertical unit vector of the world absolute dot product Select the column index corresponding to the maximum absolute dot product as the index closest to the vertical axis. Determine the sign coefficient The first of the new shaft system Column assignment value ; S4. Orthogonalize and preserve the right-handed system: Subtract the world vertical unit vector from the remaining two axis vectors. The projected components are normalized; the third axis is generated using the vector cross product operation, making the three axes pairwise orthogonal, and an intermediate rotation matrix that satisfies the right-hand coordinate system rule is constructed. S5. Align the longest edge with the local horizontal first axis: Project the point set P onto the local coordinate system corresponding to the intermediate rotation matrix, and recalculate the bounding box dimensions in each coordinate axis direction; determine if the longest dimension is on the local horizontal first axis; adjust the axis order while keeping the vertical axis constraint unchanged, and update the constrained rotation matrix R. c Constraint bounding box center c c Constraint bounding box size e c Complete the construction of the constrained bounding box; S6. Generate the end-effector pose: Using the object pose output by the constraint-oriented bounding box as a reference, select preset RPY pose candidates according to the current working area of ​​the robotic arm, and superimpose the grasping approach offset, grasping position offset, and lifting offset to calculate the end-effector grasping pose of the robotic arm. S7. Execute the gripping sequence: Control the robotic arm to move sequentially to the starting point, approach position, and gripping position, close the gripper, complete the fine-tuning action after the gripper closes, perform the lifting action, and return to the safe position; if the segmentation or motion planning fails, disturb the torso and waist joints and return to step S1 to re-execute, and set the maximum number of retries.

[0013] Furthermore, step S4 specifically includes: Subtract the vector at the other two axes besides the vertical axis from the vector at the vertical axis. The projection components in the direction are normalized and normalized to obtain the vector. Based on the already determined vertical axis and one of the horizontal axes, the third coordinate axis vector is solved by the three-dimensional vector cross product operation to ensure that the three axes are pairwise orthogonal and satisfy the right-hand coordinate system rule, thus obtaining the intermediate rotation matrix.

[0014] Furthermore, in step S5, the vertical axis direction remains fixed, and only the two horizontal coordinate axes are sequentially swapped. After the swap is completed, the center cc and dimension e of the constraint-oriented bounding box are recalculated. c .

[0015] Furthermore, in step S6, the preset RPY attitude candidates are a partitioned fixed RPY candidate library; based on the working sub-region where the robotic arm end effector is located in the arm base coordinate system, the corresponding roll, pitch, and yaw parameters are read from the RPY attitude candidate library.

[0016] Furthermore, in step S7, if the number of retries reaches the maximum retry limit, the crawling process is terminated and a crawling failure flag is output.

[0017] According to a second aspect of the present invention, a constraint-oriented bounding box gripping system for cylindrical products is provided, for implementing the above-described constraint-oriented bounding box gripping method for cylindrical products, comprising: Point cloud acquisition module, constraint OBB module, pose grabbing module, execution module; The point cloud acquisition module is used to obtain the instance mask of the target columnar product through visual retrieval and instance segmentation, generate the target point cloud by combining depth information, transform the target point cloud to the coordinate system of the robotic arm base, and output the point set P; The constrained OBB module is connected to the point cloud acquisition module; it receives the point set P, performs initial OBB solving, forces alignment of the near-vertical axis with the world Z-axis, orthogonalizes while preserving the right-hand rule, aligns the longest side with the local horizontal first axis, recalculates the bounding box center and extent, and outputs the constraint rotation matrix R corresponding to the constrained oriented bounding box. c Constraint bounding box center c c Constraint bounding box size e c ; The upper gripping pose module is connected to the constraint OBB module; it receives the object pose output by the constraint OBB, reads the RPY candidate library and gripping offset configuration parameters, and generates the upper gripping end pose of the robotic arm. The execution module is connected to the upper grasping pose module; it receives the upper grasping end pose and executes a complete action sequence of approach-grab-lift; when segmentation or planning fails, it outputs a waist disturbance command, triggers the system to call the point cloud acquisition module again, and counts the number of retries.

[0018] Furthermore, the execution module is configured with a waist disturbance retry unit; the waist disturbance retry unit is set with a maximum retry threshold, and when the number of retries exceeds the threshold, retries are stopped and a capture failure signal is output.

[0019] Furthermore, the system also stores a set of configuration parameters; the set of configuration parameters includes the upper grasp offset configuration item, RPY candidate library parameters, and waist perturbation parameters; the upper grasp offset configuration item includes the pre grasp offset, in grasp offset, and up grasp offset.

[0020] According to a third aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the constraint-oriented bounding box grasping method for columnar goods as described above.

[0021] According to a fourth aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the above-described constraint-oriented bounding box grasping method for columnar goods.

[0022] According to a fifth aspect of the present invention, a robot is provided, including a robotic arm, a torso and waist joint, a vision sensor, and the aforementioned constraint-oriented bounding box gripping system for cylindrical goods.

[0023] According to the present invention, at least the following beneficial effects are achieved: By constructing a constrained OBB using geometric constraints, the near-vertical axis is forced to align with the world vertical axis, using an orthogonal right-handed system, and the longest side is aligned with the first horizontal axis. This effectively suppresses point cloud noise and incomplete point cloud data that cause OBB principal axis tilting and jitter. The gripping posture on columnar products is more stable, reducing the probability of gripper collisions with shelves, product knocking over, and slippage. It does not require a large number of training samples or training the gripping network for every type of columnar SKU.

[0024] This invention does not require collecting massive amounts of labeled SKU samples to train a deep learning crawling model; for newly launched columnar products, as long as the products are placed vertically on the shelf, the algorithm of this invention can be used directly without retraining and parameter tuning, adapting to the business scenario of rapid iteration of massive SKUs on retail shelves.

[0025] This invention pertains to grasping geometry layer capabilities, which are orthogonal to the upper-level shelf task scheduling system. The upper-level scheduling system is responsible for product task allocation, while this invention is responsible for single-object grasping geometry calculations. The capabilities can be deployed and protected independently, and the modules are pluggable, facilitating integration into different robotic shelf picking systems.

[0026] This invention improves the robustness of the system in occluded scenarios by adding a waist perturbation retry mechanism. When visual segmentation fails due to viewpoint occlusion, the perturbation waist changes the camera viewpoint and retryes, thus improving the system's robustness in occluded scenarios. A maximum number of retries is set to avoid infinite loops.

[0027] This invention uses the RPY partitioning candidate library to select the end effector posture based on the working area of ​​the robotic arm, which can avoid singular configurations and joint limitations of the robotic arm, and can take into account both grasping geometric constraints and robotic arm kinematic constraints. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the overall process of the constraint-oriented bounding box gripping method for columnar commodities according to the present invention; Figure 2 This is a schematic diagram of the module of the constraint-oriented bounding box gripping system for columnar commodities according to the present invention. Detailed Implementation

[0029] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment uses column-shaped beverage bottles and cans as the target objects for column-shaped goods, and is applied to the top-to-bottom grabbing operation of a supermarket shelf robot. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0030] This invention addresses the scenario of top-to-bottom grasping of cylindrical bottles and cans on robotic shelves, proposing a constraint-oriented bounding box (OBB) grasping method. This invention does not rely on deep learning networks, but rather on geometric constraints to correct the original OBB.

[0031] This invention utilizes the prior knowledge of shelf-shaped product placement, where products are placed upright on shelves, and the object's axis should be close to the world's vertical Z-axis. Therefore, three layers of geometric constraints are applied to the original OBB obtained from the segmented point cloud: first, the near-vertical principal axis is forcibly aligned with the world's vertical Z-axis; second, all coordinate axes are orthogonalized while maintaining a right-handed coordinate system; and third, the vertical axis is kept stationary while aligning the longest side of the bounding box with the local horizontal first axis. This results in a constrained OBB. The constrained OBB outputs the corrected object pose, which is then combined with the RPY candidate library for the robotic arm's work zone, overlaid with grasping offset parameters, to calculate the robotic arm's end effector grasping pose. Afterward, a complete approach-grasp-lift action sequence is executed; and retrying with waist perturbations is supported when segmentation fails.

[0032] like Figure 2 As shown, the system of the present invention is divided into four main modules: point cloud acquisition module, constraint OBB module, pose grabbing module, and execution module.

[0033] Point cloud acquisition module: Receives images and depth maps from the vision sensor, performs instance segmentation, outputs the point cloud of the target product, and performs coordinate system transformation to the coordinate system of the robotic arm base.

[0034] The constrained OBB module is the core module of this invention. It receives the point cloud, performs initial OBB calculation, forces alignment of the near-vertical axis to the world Z-axis, orthogonalizes to preserve the right-handed system, aligns the longest side to the local horizontal first axis, recalculates the bounding box center and extent, and outputs the constrained OBB rotation matrix R. c Center C c Size e c .

[0035] The upper gripping pose module receives the constrained OBB object pose, reads the RPY candidate library and gripping offset configuration, and generates the upper gripping pose of the robotic arm end effector.

[0036] Execution module: Receives the end-effector's grasping pose, outputs motion commands to drive the robotic arm, and executes a complete grasping action sequence; if segmentation or motion planning fails, it triggers a lumbar joint disturbance, returns to the point cloud acquisition module to retry, and sets the maximum number of retries.

[0037] like Figure 1 As shown in the figure, the constraint-oriented bounding box gripping method for columnar goods in a specific embodiment of the present invention includes the following steps: S1. Obtain the target point cloud: The robot's vision sensor collects images of the shelf scene, and obtains the instance mask of the target columnar beverage product through visual instance segmentation. Combined with the depth map, the original point cloud of the target is generated. The coordinates of the original point cloud are transformed to the coordinate system of the robot arm base to obtain the point set P.

[0038] In a preferred embodiment, the robot is equipped with a 3D vision sensor, mounted on the robot body or the end effector of the robotic arm. The vision sensor acquires RGB images and depth maps of the shelf scene. An instance segmentation network is used to segment the images into instances, identify target columnar products, and output a target instance mask. The instance mask is a binary image; pixels within the mask belong to the target product, and pixels outside the mask belong to the background. Combined with the depth map, the depth of the corresponding pixels in the mask is extracted to generate the original point cloud of the target. The original point cloud coordinates are in the camera coordinate system. The robot system needs to perform a coordinate transformation to convert the entire point cloud to the robotic arm base coordinate system (arm base system), obtaining a point set P. The arm base coordinate system is the reference coordinate system used for robotic arm motion planning. All robotic arm end effector poses are defined in the arm base system. After the point cloud is transformed to the arm base system, subsequent OBB calculations and pose calculations can be directly interfaced with the robotic arm motion planning module.

[0039] In practical engineering, point set P may contain noisy points and have local point cloud gaps. For example, the back of a bottle or can may be pressed against the back panel of a shelf, resulting in a lack of depth points; reflections from the can may cause depth inaccuracies; and the segmentation mask may introduce some background noise from the shelf. The subsequent OBB constraint step in this invention is to perform pose correction on this noisy point set.

[0040] S2. Calculate the initial orientation bounding box: For the point set P that has been transformed to the arm base coordinate system, calculate the unconstrained initial orientation bounding box OBB, and solve for the initial rotation matrix R, the bounding box center c, and the dimension vector e. The three column vectors r1, r2, and r3 of the initial rotation matrix R represent the three principal axis directions of the initial OBB. If there is noise in the point cloud, the vertical principal axis of the initial OBB will be tilted.

[0041] The input is a point set P in the arm-based coordinate system, and the standard OBB solution algorithm is executed. The point cloud covariance matrix is ​​calculated, and its eigenvalues ​​are decomposed to obtain three eigenvectors, forming the initial rotation matrix R. These eigenvectors correspond to the three principal axes of the OBB, and each column of matrix R represents a principal axis direction vector. The entire point cloud is projected onto the three principal axis directions, the projection interval is calculated, and the bounding box center c and dimension vector e are determined. Each component of the dimension vector e represents the extent of each principal axis direction of the OBB. The resulting initial OBB is unconstrained. When noise exists in the point cloud, the principal axes of this OBB will deflect; for example, the vertical principal axis of the initial OBB, originally placed vertically like a cylindrical bottle, will deviate from the world Z-axis. Subsequent steps in this invention involve applying geometric constraints to the initial OBB to correct its orientation.

[0042] S3. Select near the vertical axis and force alignment with the world vertical direction: Set the world vertical unit vector. Traverse all column vectors of the initial rotation matrix R and calculate the vector of each principal axis. Vertical unit vector of the world absolute dot product Find the column index corresponding to the largest absolute dot product. denoted as the index closest to the vertical axis. Calculate the sign coefficient. The sign coefficient ensures that the axis direction is aligned with the world vertical axis. The newly constructed axis system's first... The column is directly assigned a value Complete near-vertical axis forced alignment with the world Z-axis.

[0043] After this step, the vertical axis of the constrained OBB is forcibly fixed as the world vertical Z-axis, eliminating the vertical axis tilt caused by the original OBB noise. However, the other two horizontal axes have not yet been processed, and the three axes may not be orthogonal or satisfy the right-hand coordinate system. Therefore, the next step, S4 orthogonalization, is required.

[0044] S4. Orthogonalize and preserve the right-handed system: Process the remaining two horizontal axis vectors, subtracting their respective world vertical vectors from each axis vector. The projection components on the vector are removed, and the vertical components are eliminated. The two vectors are normalized to unit. Based on the determined vertical axis and one of the horizontal axes, a third coordinate axis is generated through the three-dimensional vector cross product operation. The three axes are forced to be pairwise orthogonal, and the whole system is kept in a right-handed coordinate system to obtain the intermediate rotation matrix.

[0045] This step takes a fixed vertical axis as input, processes the remaining two horizontal vectors, performs orthogonal normalization to ensure that the three axes are pairwise orthogonal, and strictly maintains the right-handed coordinate system.

[0046] Subtract the vertical vector of the remaining two axis vectors from the vector in the world vector. The projection components are removed, and the two vectors fall entirely on the horizontal plane, no longer containing vertical components. Then, both vectors are normalized to unit. After deprojection and normalization, two horizontal vectors are obtained. However, the two vectors are not necessarily orthogonal. An orthogonal coordinate axis is generated using the cross product operation. Given the vertical axis vector z... w The first vector, along with one of the processed horizontal vectors, is used to obtain a third vector through a 3D vector cross product. The cross product naturally guarantees pairwise orthogonality of the three axes and strictly adheres to the right-hand coordinate system rule. This generates an intermediate rotation matrix. The intermediate rotation matrix already satisfies the following: the vertical axis equals the world Z-axis; the three axes are orthogonal and uniform; and it follows a right-hand coordinate system. However, at this point, the two horizontal axes have not yet been aligned by their longest sides, so we proceed to step S5.

[0047] S5. Align the longest edge with the local horizontal first axis: Project the original point set P onto the local coordinate system corresponding to the intermediate rotation matrix, and recalculate the dimensions of each coordinate axis of the bounding box. Determine if the maximum dimension falls on the local horizontal first axis; if the longest dimension is not on the local horizontal first axis, keep the vertical axis direction unchanged, and swap the order of the two horizontal coordinate axes to align the longest edge with the local horizontal first axis.

[0048] If the longest dimension is already on the local horizontal first axis, there is no need to change the order of the horizontal axes. If the longest dimension is on the second horizontal axis, then the order of the two horizontal coordinate axes should be swapped while keeping the vertical axis direction completely unchanged. Swapping the order of the horizontal axes aligns the longest side of the bounding box with the local horizontal first axis. The cross-section of cylindrical products is generally circular or nearly circular, while some daily chemical products have elliptical cross-sections. After orienting the horizontal axes, the first horizontal axis of the object's local coordinate system represents the object's longest horizontal direction. When generating the gripper posture subsequently, the gripper opening and closing direction can match the direction of the longest side, resulting in optimal gripping performance.

[0049] S6. Recalculate Center and Extent: After the horizontal axis order is adjusted, recalculate the bounding box center c of the constrained orientation bounding box. c Box dimensions e c Update the rotation matrix to obtain the constrained rotation matrix R. c This completes the entire construction process of the constraint OBB. The constraint OBB has eliminated noise-induced vertical axis tilt, the vertical axis is strictly aligned with the world Z-axis, the three axes are orthogonal right-handed, and the longest side is aligned with the local horizontal first axis.

[0050] S7. Generate the end effector pose: Based on the object pose output by the constrained OBB, read the RPY candidate library pre-stored by the system; select the RPY roll, pitch, and yaw attitudes corresponding to the working sub-region of the robot arm end effector in the arm base coordinate system; read the end effector offset configuration, including pre-grasp offset, in-grasp offset, and up-grasp offset; and calculate the end effector grasping pose by superimposing the offsets.

[0051] Constrain OBB to output object pose (R) c c c () represents the coordinate system of the target cylindrical product object. In the top-grabbing operation, the gripper falls from directly above the object to grasp it. However, the RPY posture of the gripper at the end of the robotic arm cannot be directly copied from the object's coordinate system posture. Due to the constraints of the robotic arm's joints and singular configurations, the appropriate RPY posture for the gripper differs depending on the same object's posture and the robotic arm's working area.

[0052] In a preferred embodiment, a partitioned fixed RPY candidate library is established. This is based on the robotic arm end effector (or the object center c). c In the working sub-region of the arm base coordinate system, a set of roll, pitch, and yaw RPY parameters are read from the RPY candidate library. The working area is pre-divided into multiple sub-regions, and each sub-region stores a set of RPY postures adapted to the robot arm's movement to avoid the robot arm entering singular positions or joints exceeding limits.

[0053] The system configuration includes three offset parameters: pre-grasp offset (approach offset), in-grasp offset (grasp position offset), and up-grasp offset (lift offset). Approach offset: The gripper approaches the object, leaving a safe distance above it. Grasp position offset: The gripper descends to the holding position. Lift offset: The distance the gripper lifts upwards after closing the grip. All offset parameters are configured primarily in the world vertical direction.

[0054] The object pose, partitioned RPY pose candidates, and grasping offset are superimposed to calculate the grasping pose at the end effector of the robotic arm. This pose is used as the input for motion planning.

[0055] S8. Execute the gripping action: The robotic arm motion controller receives the end-effector gripping posture and executes the complete action sequence: move to the starting point, move along the approach direction to the approach position, continue moving to the gripping position; drive the gripper to close; after the gripper closes, perform minor adjustments to lift the target product upwards, and then the robotic arm returns to the safe position.

[0056] In the event of segmentation failure or motion planning failure, the execution module outputs instructions to perturb the lumbar joints of the torso, and then returns to step S1 to reacquire the target point cloud for retry; the system pre-sets a maximum retry limit, and when the retry limit is reached, the grasping task is terminated and a grasping failure flag is output.

[0057] In a preferred embodiment, the execution module receives the end-effector grasping pose and drives the robotic arm to execute a complete grasping motion sequence: Step 1: The robotic arm moves to the starting waypoint (safe waypoint); Step 2: Move to the pre-grasp position; Step 3: Continue moving downwards to the grasp position; Step 4: Output command to close the gripper; Step 5: After the grippers close, perform minor adjustments to eliminate residual positional errors; Step 6: Raise the target product upwards until it reaches the raised position; Step 7: The robotic arm moves back to the safe position.

[0058] During the operation, instance segmentation may fail, resulting in no target point cloud output; or motion planning may fail, preventing the robot arm from planning its trajectory. In the event of these failures, the execution module triggers a waist perturbation retry mechanism: it outputs control commands to perturb the robot's waist joints, rotating the waist by a small angle to change the viewing angle of the vision sensor. After the perturbation is complete, the process returns to S1 to acquire the target point cloud and re-executes the entire constrained OBB grasping process. The system pre-sets a maximum retry limit, for example, 3 times. When the retry limit is reached, retries cease, the grasping task is terminated, a grasping failure flag is output, and the upper-level scheduling system is notified.

[0059] Visual segmentation failures often stem from viewpoint occlusion. A small waist rotation changes the camera's viewpoint, mitigating occlusion and potentially restoring normal segmentation. This retry mechanism improves the overall robustness of the system.

[0060] In another specific embodiment, a constraint-oriented bounding box grasping system for columnar goods is also provided to implement the above method. The system includes a point cloud acquisition module, a constraint OBB module, a grasping pose module, and an execution module.

[0061] The point cloud acquisition module receives the image and depth data output by the vision sensor, completes instance segmentation, generates the target point cloud and performs coordinate system transformation, and outputs the point set P in the arm base coordinate system.

[0062] In a preferred embodiment, the input is an RGB image and a depth map; the output is a target point set P in the arm base coordinate system. An instance segmentation algorithm is run to generate a target instance mask; the mask is combined with the depth map to generate the original point cloud in the camera coordinate system; hand-eye calibration parameters are read, coordinate transformation is performed, and the point cloud is converted to the robotic arm base coordinate system, outputting the point set P. If segmentation fails, an empty point cloud marker is output.

[0063] The constrained OBB module receives a point set P, sequentially performs the following steps: initial OBB solution, forced alignment of the near-vertical axis to the world Z-axis, orthogonalization to preserve the right-handed system, alignment of the longest side to the local horizontal first axis, recalculation of the bounding box center and extent, and outputs the constrained OBB R. c c c e c .

[0064] In a preferred implementation, the internal sub-processes are executed sequentially: 1) Solving for the initial OBB; 2) Select the closest vertical axis and force alignment to world Z; 3) Orthogonalize and preserve the right-handed system; 4) Align the longest side with the local horizontal first axis; 5) Recalculation center and extent.

[0065] If the input point cloud is valid, output constraint OBB; if the point cloud is empty, output a failure flag.

[0066] The upper gripping pose module receives the object pose output by the constraint OBB, reads the RPY candidate library and offset configuration parameters, and calculates the upper gripping pose of the robotic arm end effector.

[0067] The execution module receives the end-effector pose and outputs motion commands to the robotic arm controller to execute approach-grab-lift; it has a built-in waist disturbance retry unit to count the number of retries and output a failure signal if the limit is exceeded.

[0068] The system stores a set of configuration parameters, which includes the grasp offset configuration items pre / in / up graspoffset, partition fixed RPY candidate library parameters, and waist perturbation parameters.

[0069] All parameters are configurable, and no modification to the algorithm code is required.

[0070] Upgrasp offset configuration options: pre-grasp offset (approach offset); in-grasp offset (grasp position offset); upgrasp offset (lift offset); unit: millimeters; controls the vertical distance of each grab point.

[0071] RPY candidate library: Roll, pitch, and yaw parameters are stored in partitions according to the working sub-region of the arm base coordinate system.

[0072] Waist disturbance parameters: range of waist rotation angle; maximum number of retries.

[0073] The present invention also provides a computer-readable storage medium for storing a computer program, which executes all the steps of the aforementioned method when the program is run; it also provides a computer device, a memory for storing the program, and a processor for running the program to implement the grasping method; and it also provides a robot equipped with a robotic arm, a torso and waist joints, and a vision sensor, equipped with the upper grasping system of the present invention to complete the top-to-bottom grasping of columnar bottles and cans on shelves.

[0074] The invention will be further described in detail below using a specific example of grabbing beverages from a shelf column (bottled beverages on a supermarket shelf).

[0075] The robot consists of a mobile chassis, a torso and waist rotation joints, a six-axis robotic arm, and an end effector gripper. A 3D depth camera is mounted on the end effector of the robotic arm. The shelf displays multiple 500mL bottles of beverages, arranged vertically, typical of columnar products. The upper-level controller issues a grabbing command to retrieve a specific bottle of beverage from the shelf.

[0076] Step 1: Point Cloud Acquisition Module. The camera acquires RGB-D images of the shelf; the instance segmentation network identifies the target beverage bottle and outputs an instance mask; a point cloud in camera coordinate system is generated by combining depth; hand-eye calibration is performed to transform to arm-based coordinate system, and the point set P is output. This point cloud contains noise; the back of the bottle is close to the back panel of the shelf, so there is no point cloud on the back; a small amount of shelf noise is mixed into the point cloud on the side.

[0077] Step 2: Constrain the OBB module to calculate the initial OBB for P. Due to point cloud incompleteness and noise, the initial OBB's vertical principal axis is tilted approximately 11 degrees relative to the world Z-axis. Proceed to step S3 to calculate the absolute values ​​of the dot products of the three principal axes of the initial OBB with the world vertical vector, and find the index of the nearest vertical axis. The sign coefficient calculation forces the axis to be replaced with the world vertical axis. S4 then orthogonalizes the remaining two horizontal axes, removes the vertical components, normalizes, and generates a right-handed orthogonal intermediate rotation matrix using a cross product. S5 projects the point cloud onto the intermediate coordinate system, compares the horizontal axis dimensions, and swaps the horizontal axis order to align the longest side with the local first horizontal axis. S6 recalculates the constraint OBB center c. c Size e c Output constraint rotation matrix R c After constraint, the vertical axis of OBB is constrained to be strictly along the world Z-axis, eliminating the original 11-degree tilt.

[0078] Step 3: Read R from the upper gripping pose module c c c Determine if the object's center falls within a working sub-region of the arm base coordinate system, and read the pre-stored RPY pose of that region; read the grasp offset parameters, and calculate the end poses of the approach position, grasp position, and lift position.

[0079] Step 4: The execution module issues motion commands. The robotic arm moves to the initial safe waypoint; moves to the approach position (80mm above the bottle); moves downward to the gripping position; the grippers close and hold the bottle; performs minor position adjustments; raises 120mm; returns to the safe point. Gripping complete.

[0080] Assuming a segmentation failure occurs in this embodiment due to the camera's view being obstructed by adjacent items, the instance segmentation outputs an empty mask, and the point cloud acquisition module outputs an empty point cloud marker. The constraint OBB module receives the empty point cloud and outputs a failure marker. The execution module detects the failure, performs a waist perturbation (rotating the waist joint by 6 degrees), increments the retry counter by 1, and returns to the point cloud acquisition module to re-acquire image segmentation. If three consecutive retries fail, reaching the maximum retry limit, the capture process stops, and a capture failure is reported to the upper-level scheduler.

[0081] This embodiment does not require any model training on the beverage bottle. It relies entirely on geometric constraints to correct the OBB pose, suppress the pose tilt caused by point cloud noise, and achieve stable upward gripping.

[0082] In summary, the present invention has been described in detail through specific embodiments. Those skilled in the art will understand that various modifications and changes can be made. For example, the present invention is mainly aimed at vertically placed cylindrical bottles and cans on shelves, including cylindrical and elliptical cylindrical objects. The present invention is not limited to beverage bottles; it is applicable to cylindrical products such as daily chemical bottles and canned foods. It can also disable waist perturbation, disable the retry function, and set the maximum number of retries to 0, making it suitable for robotic arms without waist rotation joints. The RPY candidate library can be simplified to a fixed set of RPYs without partitioning, suitable for scenarios where the robotic arm's workspace is small and the risk of singularities is low. The offset parameter can be determined based on the constrained OBB size e. c Adaptive calculations, such as automatically calculating the grab height based on the bounding box height, do not require a completely fixed configuration. Such modifications and variations, as long as they do not depart from the spirit and intent of this invention, shall fall within the scope of protection of this invention, which is defined by the appended claims.

Claims

1. A method for gripping a bounding box with constraint orientation for columnar goods, characterized in that, Includes the following steps: S1. Obtain the target point cloud: Obtain the visual instance mask of the target columnar product, combine it with the depth information to generate the target point cloud, transform the target point cloud to the coordinate system of the robotic arm base, and obtain the point set P; S2. Calculate the initial orientation bounding box: Solve for the initial orientation bounding box of the point set P to obtain the initial rotation matrix R, the bounding box center c, and the size vector e; the three columns of the initial rotation matrix R correspond to the three principal axis directions of the initial OBB. S3. Select the nearest vertical axis and force alignment with the world vertical direction; set the world vertical unit vector. ; Calculate the principal axis vector of each column of the initial rotation matrix R. Vertical unit vector of the world absolute dot product Select the column index corresponding to the maximum absolute dot product as the index closest to the vertical axis. Determine the sign coefficient The first of the new shaft system Column assignment value ; S4. Orthogonalize and preserve the right-handed system: Subtract the world vertical unit vector from the remaining two axis vectors. The projected components are normalized; the third axis is generated using the vector cross product operation, making the three axes pairwise orthogonal, and an intermediate rotation matrix that satisfies the right-hand coordinate system rule is constructed. S5. Align the longest edge with the local horizontal first axis: Project the point set P onto the local coordinate system corresponding to the intermediate rotation matrix, and recalculate the bounding box dimensions in each coordinate axis direction; determine if the longest dimension is on the local horizontal first axis; adjust the axis order while keeping the vertical axis constraint unchanged, and update the constrained rotation matrix R. c Constraint bounding box center c c Constraint bounding box size e c Complete the construction of the constrained bounding box; S6. Generate the end-effector pose: Using the object pose output by the constraint-oriented bounding box as a reference, select preset RPY pose candidates according to the current working area of ​​the robotic arm, and superimpose the grasping approach offset, grasping position offset, and lifting offset to calculate the end-effector grasping pose of the robotic arm. S7. Execute the gripping sequence: Control the robotic arm to move sequentially to the starting point, approach position, and gripping position, close the gripper, complete the fine-tuning action after the gripper closes, perform the lifting action, and return to the safe position; if the segmentation or motion planning fails, disturb the torso and waist joints and return to step S1 to re-execute, and set the maximum number of retries.

2. The constraint-oriented bounding box gripping method for columnar goods according to claim 1, characterized in that, Step S4 specifically includes: subtracting the vector at the other two axes (excluding the vertical axis) from the vector at the vertical axis. The projection components in the direction are normalized and normalized to obtain the vector. Based on the already determined vertical axis and one of the horizontal axes, the third coordinate axis vector is solved by the three-dimensional vector cross product operation to ensure that the three axes are pairwise orthogonal and satisfy the right-hand coordinate system rule, thus obtaining the intermediate rotation matrix.

3. The constraint-oriented bounding box gripping method for columnar goods according to claim 1 or 2, characterized in that, In step S5, the vertical axis direction remains fixed, and only the two horizontal coordinate axes are sequentially swapped. After the swap, the center c of the constraint-oriented bounding box is recalculated. c With size e c .

4. The constraint-oriented bounding box gripping method for columnar goods according to claim 3, characterized in that, In step S6, the preset RPY attitude candidates are a partitioned fixed RPY candidate library; based on the working sub-region where the end effector of the robotic arm is located in the arm base coordinate system, the corresponding roll, pitch, and yaw parameters are read from the RPY attitude candidate library.

5. The constraint-oriented bounding box gripping method for columnar goods according to claim 4, characterized in that, In step S7, if the number of retries reaches the maximum number of retries, the crawling process is terminated and a crawling failure flag is output.

6. A constraint-oriented bounding box gripping system for columnar goods, used to implement the constraint-oriented bounding box gripping method for columnar goods according to any one of claims 1-5, characterized in that, include: Point cloud acquisition module, constraint OBB module, pose grabbing module, execution module; The point cloud acquisition module is used to obtain the instance mask of the target columnar product through visual retrieval and instance segmentation, generate the target point cloud by combining depth information, transform the target point cloud to the coordinate system of the robotic arm base, and output the point set P; The constrained OBB module, connected to the point cloud acquisition module, receives the point set P, performs initial OBB solving, forces alignment of the near-vertical axis with the world Z-axis, orthogonalizes while preserving the right-hand rule, aligns the longest side with the local horizontal first axis, recalculates the bounding box center and extent, and outputs the constraint rotation matrix R corresponding to the constrained oriented bounding box. c Constraint bounding box center c c Constraint bounding box size e c ; The upper gripping pose module is connected to the constraint OBB module, receives the object pose output by the constraint OBB, reads the RPY candidate library and gripping offset configuration parameters, and generates the upper gripping end pose of the robotic arm. The execution module is connected to the upper-grabbing pose module, receives the upper-grabbing end pose, and outputs a waist perturbation command when segmentation or planning fails, triggering the system to call the point cloud acquisition module again and counting the number of retries.

7. The constraint-oriented bounding box gripping system for columnar goods according to claim 6, characterized in that, The execution module is configured with a waist disturbance retry unit; the waist disturbance retry unit is set with a maximum retry threshold. When the number of retries exceeds the threshold, retries are stopped and a capture failure signal is output.

8. The constraint-oriented bounding box gripping system for columnar goods according to claim 6 or 7, characterized in that, The system also stores a set of configuration parameters; the set of configuration parameters includes the upper grasp offset configuration item, RPY candidate library parameters, and waist perturbation parameters; the upper grasp offset configuration item includes the pre grasp offset, in grasp offset, and up grasp offset.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the constraint-oriented bounding box grasping method for columnar commodities as described in any one of claims 1-5.

10. A robot, characterized in that, It includes a robotic arm, a torso and waist joint, a vision sensor, and a constraint-oriented bounding box gripping system for cylindrical goods as described in any one of claims 6-8.

Images