Robot loading and unloading control method and device based on visual guidance and robot

Abstract

The application discloses a kind of based on visual guidance's robot loading and unloading control method, device and robot, control method includes: obtaining the three-dimensional information of loading and unloading area that global vision sensor gathers and identifying each cargo and its pose;According to the pose of each cargo, carry out the decision of grabbing, determine target cargo and theoretical grabbing pose;In the process that mobile platform advances towards target cargo, according to the pose of mobile platform and the pose of target cargo update relative pose and adjust the posture of mechanical arm;When reaching target area, control end vision sensor local imaging and compensate theoretical grabbing pose, generate final grabbing pose after control mechanical arm grabs.The present application carries out wide-range perception and grabbing planning by global vision, carries out local accurate imaging and pose compensation by end vision, can complete whole process from target identification to posture pre-adjustment in the process of moving, after reaching target area, local compensation can be executed accurate grabbing, greatly improve the overall efficiency of loading and unloading operation.

Description

Technical Field

[0001] This invention relates to the field of intelligent warehousing and logistics and robot control technology, and in particular to a vision-guided robot loading and unloading control method, device and robot. Background Technology

[0002] In warehousing and logistics, loading, unloading, and palletizing at loading docks are core logistics processes. Composite robots have been introduced into these processes to replace manual labor in loading and unloading operations.

[0003] The composite robot mounts a robotic arm on an autonomous mobile platform, which enables the robotic arm to move flexibly between the platform and the truck, thus extending the robotic arm's workspace to the entire loading and unloading area. For visual guidance of the composite robot, a multi-camera vision-guided stacking posture planning method is typically used. This method involves using multiple cameras positioned at different locations in the loading and unloading area to acquire multi-view images of the stacked goods, fusing them to generate three-dimensional pose information of the goods, and then planning the placement order and pose of the goods accordingly.

[0004] However, the above-mentioned existing technical solutions still have the following shortcomings: Firstly, existing solutions mainly focus on solving the pose planning problem during the cargo placement stage, while they lack coverage for dynamic planning and precise control of how to effectively grab cargo from randomly stacked goods. Secondly, in the existing vision-guided composite robots, the mobile platform first travels to the target area and comes to a complete stop, then the vision system is activated to scan and identify. Only after all perception and planning are completed does the robotic arm begin to perform grasping, which leads to a longer work cycle and affects overall efficiency. Summary of the Invention

[0005] This invention provides a vision-guided robot loading and unloading control method, device, and robot to solve the problems in the prior art where robots have long operation cycles due to the use of a stop-then-grab operation mode, and where the focus is on placement posture planning but lacks dynamic planning and precise control of the gripping end under random stacking conditions.

[0006] In a first aspect, embodiments of the present invention provide a vision-guided robot loading and unloading control method, applied to a robot, the robot including a mobile platform, a global vision sensor, a robotic arm disposed on the mobile platform, and an end-effector vision sensor disposed at the end of the robotic arm, the control method including: The three-dimensional information of the loading and unloading area collected by the global vision sensor is obtained, and each cargo and its pose are identified based on the three-dimensional information. Based on the pose of each item, a grasping decision is made to determine the target item and the theoretical grasping pose; During the process of the mobile platform moving towards the target cargo, the relative pose of the target cargo with respect to the robotic arm is updated according to the real-time pose of the mobile platform and the pose of the target cargo, and the posture of the robotic arm is adjusted according to the relative pose. When the mobile platform reaches the target area, it controls the end vision sensor to perform local imaging of the target cargo, and compensates the theoretical grasping pose based on the local imaging data to generate the final grasping pose. The robotic arm is controlled to grasp the target cargo based on the final grasping pose.

[0007] Secondly, embodiments of the present invention provide a vision-guided robot loading and unloading control device, applied to a robot, the robot including a mobile platform, a global vision sensor, a robotic arm disposed on the mobile platform, and an end-effector vision sensor disposed at the end of the robotic arm, including: The cargo identification module is used to acquire the three-dimensional information of the loading and unloading area collected by the global vision sensor, and to identify each cargo and its pose based on the three-dimensional information. The theoretical grasping pose determination module is used to make grasping decisions based on the poses of each of the goods, and to determine the target goods and the theoretical grasping pose; The robotic arm posture adjustment module is used to update the relative posture of the target cargo with respect to the robotic arm based on the real-time posture of the mobile platform and the posture of the target cargo during the process of the mobile platform moving towards the target cargo, and to adjust the posture of the robotic arm based on the relative posture. The final grasping pose generation module is used to control the end vision sensor to perform local imaging of the target cargo when the mobile platform reaches the target area, and to compensate the theoretical grasping pose based on the local imaging data to generate the final grasping pose. The grasping control module is used to control the robotic arm to grasp the target goods according to the final grasping pose.

[0008] Thirdly, embodiments of the present invention provide a bin robot, the bin robot comprising: Mobile platform; A robotic arm mounted on the mobile platform and an end vision sensor mounted at the end of the robotic arm; Global vision sensor; One or more controllers; Storage device for storing one or more programs; The mobile platform, the global vision sensor, and the end vision sensor all communicate with the controller. When the one or more programs are executed by the one or more controllers, the one or more controllers implement the vision-guided robot loading and unloading control method described in the first aspect.

[0009] This invention identifies each cargo and its pose by acquiring 3D information of the loading / unloading area from a global vision sensor, makes grasping decisions to determine the target cargo and theoretical grasping pose, and continuously updates the relative pose of the target cargo with respect to the robotic arm while the mobile platform is moving, and adjusts the robotic arm's posture in real time. This achieves a parallel operation mode of planning while moving, making full use of the mobile platform's travel time window and significantly shortening the grasping cycle of a single cargo. Simultaneously, global vision enables large-scale perception and grasping planning, while end-effector vision performs precise local imaging and pose compensation, ensuring both the global optimality of the grasping decision and the local accuracy of the final grasping pose. This allows the robot to complete the entire process from target recognition to posture pre-adjustment during movement, and only requires local compensation to perform grasping upon reaching the target area, significantly improving the overall efficiency of loading / unloading operations. Furthermore, by continuously updating the relative pose based on the real-time pose of the mobile platform and the target cargo during movement, the influence of platform movement on target positioning accuracy is eliminated, overcoming the technical bias of traditional solutions that require a stationary state to accurately obtain the target pose. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the robot according to an embodiment of the present invention; Figure 2 This is a flowchart of a vision-guided robot loading and unloading control method provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the determination of the target cargo and the theoretical grasping pose in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the dynamic adjustment of the robotic arm's pose in an embodiment of the present invention; Figure 5 This is a flowchart of generating the final grasping pose in an embodiment of the present invention; Figure 6 This is a schematic diagram of a vision-guided robot loading and unloading control device provided in an embodiment of the present invention. Detailed Implementation

[0011] To make the technical problems solved by the present invention, the technical solutions adopted, and the technical effects achieved clearer, the technical solutions of the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] like Figure 1 As shown, the robot in this embodiment of the invention includes a mobile platform 100, a robotic arm 200 mounted on the mobile platform 100, an end effector 210 and an end vision sensor 220 mounted at the end of the robotic arm 200, and a global vision sensor 300 mounted at a high position in the loading and unloading scene (such as the top of the platform or a column). Multiple goods 400 are stacked in the loading and unloading area, including various packaging forms such as cardboard boxes, burlap sacks, and containers. The mobile platform 100 travels on the platform floor, moving from its starting position towards the target area where the target goods 400 are located. For example, in one scenario, goods are unloaded from a truck onto the platform in a stacked manner, and the robot needs to remove the goods from the stacks and transport them to a warehouse for storage.

[0013] Figure 2 A flowchart of a vision-guided robot loading and unloading control method provided in an embodiment of the present invention is shown below. Figure 2 As shown, the vision-guided robot loading and unloading control method specifically includes the following steps: S10. Obtain the three-dimensional information of the loading and unloading area collected by the global vision sensor, and identify each cargo and its pose based on the three-dimensional information.

[0014] A global vision sensor 300 (e.g., a wide-field-of-view 3D camera) can be fixedly installed on the top of the platform or high up in the loading and unloading area, allowing it to overlook the entire area and collect 3D point cloud data containing all stacked goods as 3D information. The global vision sensor 300 can wirelessly transmit this 3D information to the robot's controller, which can be installed in the mobile platform. Since the installation position and orientation of the global vision sensor are known after calibration, the collected point cloud data is located in the global coordinate system. After acquiring this 3D information, the robot controller uses a point cloud processing algorithm to identify each individual item from the point cloud and calculates the pose of each item in the global coordinate system. This pose includes the item's 3D position coordinates (x, y, z) and orientation angles (roll angle, pitch angle, yaw angle) in the global coordinate system, providing basic data for subsequent grasping decisions.

[0015] In one embodiment, step S10 may include the following sub-steps: S101 can control the global vision sensor to perform a global scan of the loading and unloading area and obtain raw global point cloud data.

[0016] The controller sends a scanning command to the global vision sensor 300. The global vision sensor 300 performs a complete 3D scan of the loading and unloading area to obtain raw point cloud data containing cargo stacks, the ground, and the surrounding environment. The raw point cloud data is a collection of points in three-dimensional space, and each point contains three-dimensional coordinate information in the global coordinate system.

[0017] S102. Filter and denoise the original global point cloud data and perform plane segmentation to extract the effective point cloud data of the cargo stacking surface in the loading and unloading area.

[0018] Because the global vision sensor may be affected by factors such as ambient light, sensor noise, and reflections from the cargo surface during scanning, the raw point cloud data inevitably contains noise points and outliers. This step first performs noise reduction processing on the raw global point cloud data, for example, using statistical filtering or radius filtering algorithms to remove isolated noise points. Subsequently, a plane segmentation algorithm (such as the Random Sample Consensus Algorithm, RANSAC) is used to identify and remove non-cargo planes such as the ground, extracting only valid point cloud data containing the cargo stacked surfaces. Valid point cloud data refers to the point cloud located on the outer surface of the cargo, excluding noise points and ground points, and is used for subsequent cargo identification and segmentation.

[0019] S103. Based on the effective point cloud data, a point cloud clustering segmentation algorithm is used to independently segment each cargo from the stacked point cloud, and a three-dimensional bounding box is fitted to each cargo point cloud cluster after independent segmentation, outputting the three-dimensional bounding box information of each cargo and its pose in the global coordinate system.

[0020] The effective point cloud data contains multiple stacked goods. The point clouds of different goods exhibit spatial clustering characteristics. This step employs a point cloud clustering segmentation algorithm (such as a clustering algorithm based on Euclidean distance or a segmentation algorithm based on region growing) to group spatially adjacent points into the same point cloud cluster and spatially separated points into different point cloud clusters. This effectively separates each goods from the stacked point cloud, with each point cloud cluster corresponding to an independent goods. Subsequently, a 3D bounding box fitting is performed on each goods point cloud cluster. This involves calculating the smallest cuboid that can enclose the cluster, obtaining the center coordinates, dimensions (length, width, height), and orientation angle of this cuboid. The center coordinates of this 3D bounding box represent the goods' position, and the orientation angle represents the goods' pose. Finally, the 3D bounding box information and pose of each goods are output.

[0021] This embodiment removes noise and outliers from the point cloud through filtering and noise reduction, ensuring the accuracy of subsequent segmentation and fitting. At the same time, it eliminates ground interference through planar segmentation, reducing the missegmentation rate. Furthermore, it adopts point cloud clustering segmentation combined with 3D bounding box fitting, which can quickly and accurately identify each independent item and its pose from the stacked point cloud, providing reliable input data for grasping decisions.

[0022] S20. Make a grasping decision based on the pose of each item, and determine the target item and the theoretical grasping pose.

[0023] After identifying the poses of each cargo in the loading and unloading area, it is necessary to select the cargo that is most suitable for being grasped in the current state as the target cargo, and determine the theoretical pose (i.e., theoretical grasping pose) that the end effector of the robotic arm should reach when grasping the cargo. This decision-making process needs to take into account the stacking relationship of the cargo (the upper cargo blocks the lower cargo, and the lower cargo is pressed down by the upper cargo and cannot be reached) and the robotic arm's own motion capability (workspace constraints).

[0024] like Figure 3 As shown, in an optional embodiment, step S20 includes the following sub-steps: S201. Calculate the support and coverage relationship between goods based on the position of each goods, and determine the stacking level of each goods.

[0025] The pose of each item includes its three-dimensional position and orientation in the global coordinate system. Combined with the three-dimensional bounding box dimensions of each item, the spatial overlap relationship between any two items can be calculated. For example, if the bottom of one item is located above the top of another item and its projection on the horizontal plane overlaps with the top projection of the other item, then the former is determined to be stacked on top of the latter. By traversing the support and coverage relationships between all items, the hierarchical structure of the entire stack can be constructed, determining the stacking level (i.e., which layer) of each item. The top-level item refers to the item that is not pressed down by any other item, i.e., there are no other items above it.

[0026] S202. Generate an initial fetch sequence according to the stacking hierarchy and the priority rules from top to bottom and from outside to inside.

[0027] In destacking operations, the grabbing decision requires grabbing the upper-layer goods first to expose the lower-layer goods. Furthermore, within the same layer, goods on the periphery are easier to grab and less likely to interfere with adjacent goods than those on the inside. Therefore, this step prioritizes all goods according to stacking level from highest to lowest (i.e., top to bottom), and within the same layer, prioritizes them according to distance from the stack center from farthest to nearest (i.e., outside to inside), generating an initial grabbing sequence. Goods listed earlier in this sequence are grabbed with higher priority.

[0028] S203. During the movement of the mobile platform, it is detected in real time whether the first item in the initial grasping sequence meets the workspace constraints of the robotic arm (i.e., whether the robotic arm can reach the item without collision in the current platform pose). If it meets the constraints, it is determined to be the target item. If it does not meet the constraints, it is postponed to the next item until the item that meets the workspace constraints is selected as the target item.

[0029] Workspace constraints refer to the spatial area within which the robotic arm's end effector can reach without collisions and whose joint angles do not exceed limits, given the current pose of the mobile platform. The initial grasping sequence is a globally optimal order generated based on stacking relationships and positional priorities. However, the actual reachability of the robotic arm is also affected by the current position of the mobile platform. Some goods may be located outside the boundaries of the robotic arm's workspace and cannot be reached by the robotic arm from the current platform position. Therefore, as the mobile platform moves towards the loading / unloading area, the controller detects the first item in the current initial grasping sequence (i.e., the current first item) in real time at a preset period (e.g., 50ms), determining whether the relative pose of this item with respect to the robotic arm base is within the robotic arm's workspace. If the workspace constraint is met, the item is identified as the target item; otherwise, it is temporarily marked as unreachable and moved to the next item in the sequence, and the detection continues until an item that meets the workspace constraint is selected as the target item.

[0030] S204. After determining the target cargo, calculate the theoretically grasped position point according to the following formula: ; in, This refers to the three-dimensional coordinates of the theoretically captured pose point in the global coordinate system, i.e., the target position that the robotic arm's end effector should reach. The geometric center coordinates of the three-dimensional bounding box of the target cargo are extracted from the cargo pose information output in step S103. d is the preset gripping offset distance, which refers to the distance that the end effector offsets from the geometric center of the cargo along the gripping direction. It is preset according to the cargo size and end effector type. For example, for a carton, it can be preset to half the cargo length plus the finger length of the end effector. The grasping direction unit vector is the approach direction when the end effector approaches the cargo.

[0031] S205. Dynamically determine the unit vector of the grabbing direction based on the stacking level of the target goods. .

[0032] If the target cargo is not a top-level cargo, the grabbing direction unit vector is selected as the horizontal direction vector; if the target cargo is a top-level cargo, the grabbing direction unit vector is selected as the weighted composite vector of the vertical and horizontal direction vectors.

[0033] Specifically, the stacking level of the target goods determines the spatial constraints around them. For goods that are not on the top layer (i.e., there are other goods above them), approaching them from above will be blocked by the goods above, so they can only be approached from the horizontal direction (i.e., the side of the goods). The unit vector for the grasping direction is chosen to be the horizontal vector. For top-layer goods (i.e., there are no other goods above them), they can be approached vertically from above or horizontally from the side. The vertical and horizontal vectors are weighted and combined to form the unit vector for the grasping direction. This can be flexibly adjusted according to the actual working conditions. For example, when there is interference around the top-layer goods, the weight of the vertical component can be increased to grasp them from top to bottom.

[0034] By constructing a stacking hierarchy and an initial grasping sequence, and by detecting workspace constraints in real time during platform movement, it is ensured that the selected target goods are actually reachable by the robotic arm at the current position. This avoids the wasted time of discovering that the target is unreachable only after the platform has reached the target area, which would require replanning. At the same time, by dynamically selecting the grasping direction based on the stacking hierarchy (not the top layer being forced horizontal, and the top layer being able to weightedly combine vertical and horizontal directions), the grasping path planning can adaptively avoid stacking interference, improving the grasping success rate and adaptability to working conditions.

[0035] S30. During the process of the mobile platform moving towards the target cargo, the relative pose of the target cargo with respect to the robotic arm is updated according to the real-time pose of the mobile platform and the pose of the target cargo, and the posture of the robotic arm is adjusted according to the relative pose.

[0036] The pose of the target cargo is described in the global coordinate system, while the motion control of the robotic arm needs to know the pose of the target cargo relative to its own base coordinate system. As the mobile platform moves towards the target area, the robotic arm base coordinate system moves accordingly due to the continuous movement of the mobile platform, and the relative pose of the target cargo relative to the robotic arm base also changes continuously. This step continuously calculates the relative pose during the platform's movement and adjusts the robotic arm's posture accordingly so that the end effector always points towards the target cargo during the movement.

[0037] like Figure 4 As shown, in an optional embodiment, step S30 includes the following sub-steps: S301. Obtain the current platform pose of the mobile platform in the global coordinate system.

[0038] The mobile platform 100 is equipped with a positioning system (such as laser SLAM, inertial measurement unit IMU or wheeled odometer). The positioning system outputs the current position and orientation of the mobile platform 100 in the global coordinate system (world coordinate system) in real time, that is, the current platform pose.

[0039] S302. Based on the pose of the target cargo and the current platform pose, calculate the relative pose using the following formula: ; in, Let be the homogeneous transformation matrix corresponding to the relative pose of the target cargo with respect to the robotic arm. The homogeneous transformation matrix corresponding to the current platform pose is a 4×4 matrix that contains the rotation matrix and translation vector of the mobile platform in the global coordinate system, representing the position and orientation of the platform. The homogeneous transformation matrix corresponding to the pose of the target cargo is also a 4×4 matrix, which contains the rotation matrix and translation vector of the target cargo in the global coordinate system, expressing the position and orientation of the cargo.

[0040] Homogeneous transformation matrices are commonly used mathematical tools in robotics to describe the position and orientation of rigid bodies. Their basic form is as follows: ; Where R is a 3×3 rotation matrix (describing the attitude) and t is a 3×1 translation vector (describing the position).

[0041] The meaning of the relative pose calculation formula is: the pose matrix of the target cargo in the coordinate system of the robot arm base is equal to the inverse matrix of the pose matrix of the mobile platform in the global coordinate system multiplied by the pose matrix of the target cargo in the global coordinate system. The principle is: first, the target cargo is transformed from the global coordinate system to the mobile platform coordinate system (by left-multiplying by the inverse matrix of the platform pose matrix). Since the robot arm base is fixed to the mobile platform, the pose obtained by the transformation is the relative pose of the target cargo with respect to the robot arm base.

[0042] S303. Based on the moving speed of the mobile platform and the time rate of change of the relative pose, perform Kalman filtering to predict the relative pose and generate a filtered smooth relative pose.

[0043] During the movement of the mobile platform, the measurements from the positioning system contain noise, and due to communication and control delays, directly using the calculation results at the current moment for robotic arm control may result in lag. This step uses the Kalman filter algorithm to filter and predict the relative pose calculated in step S302.

[0044] Specifically, the moving speed of the mobile platform is used as the control input of the state transition model, and the time change rate of the relative pose is used as the rate of change of the state variable to establish the state equation for the relative pose. Using the relative pose calculated in step S302 as the observation value, the Kalman filter prediction and update steps are executed to generate a filtered, smoothed relative pose. Kalman filtering is an optimal estimation algorithm suitable for linear Gaussian systems. By fusing the system model prediction and noisy observation values, the optimal estimate in the sense of minimum mean square error is obtained. This embodiment uses Kalman filtering to predict and smooth the relative pose, eliminating the influence of positioning measurement noise and control delay, and ensuring the smoothness of the robotic arm's movement and tracking accuracy.

[0045] S304. Based on the relative pose and combined with the current angle information of each joint of the robotic arm, solve the desired joint angle combination of the robotic arm at the entrance of the target area through inverse kinematics, so that the end effector of the robotic arm points to the target cargo.

[0046] After obtaining the filtered and smoothed relative pose, the robotic arm controller uses this relative pose as the target and combines it with the actual angles of each joint of the robotic arm at present. Through inverse kinematics calculation, it calculates the desired combination of angles of each joint of the robotic arm at the entrance of the target area (i.e., when the mobile platform has just entered the target area and has not yet completely stopped) so that the position and orientation of the end effector in space are aligned with the target cargo. Inverse kinematics refers to the process of solving the angles of each joint of the robotic arm when the desired pose of the end effector in space is known. It is a basic method of robotic arm control.

[0047] S305. During the process of the mobile platform moving from its current position to the target area, the inverse kinematics solution is repeatedly executed at preset time intervals to generate continuous joint trajectories of each joint of the robotic arm, and the robotic arm is controlled to move according to the continuous joint trajectories so that the end effector can complete the pre-alignment of the target goods before reaching the target area.

[0048] As the mobile platform continues to move, the relative pose of the target cargo with respect to the robotic arm base constantly changes. Therefore, steps S302 to S304 need to be repeated at preset time intervals (e.g., 10ms or 20ms) throughout the entire process of the platform moving from its current position to the target area. The joint angle combinations obtained each time constitute a joint angle sequence. The joint angle sequences at all times are spliced ​​together to form the continuous joint trajectory of each joint of the robotic arm. The controller controls each joint of the robotic arm to move according to this continuous joint trajectory, so that the end effector continuously tracks the pose change of the target cargo during the movement of the mobile platform. When the platform arrives at the target area, the end effector has completed the pre-alignment of the target cargo (i.e., pointing towards the target cargo and the opening, orientation, etc. of the end effector have been adjusted to be close to the grasping posture).

[0049] By continuously calculating the relative pose of the target cargo with respect to the robotic arm base during the movement of the mobile platform, the grasping preparation process and the movement process are executed in parallel. When the platform arrives at the target area, the robotic arm has already completed pre-alignment, eliminating the need to wait for the platform to come to a complete stop before planning begins, thus significantly shortening the operation cycle. In addition, by repeatedly solving inverse kinematics at preset time intervals to generate continuous joint trajectories, the movement of the robotic arm is synchronized with the movement of the platform, realizing a continuous operation mode of moving and adjusting simultaneously. This avoids large-scale rapid start-stop movements, reducing energy consumption and mechanical wear.

[0050] S40. When the mobile platform reaches the target area, the control end vision sensor performs local imaging of the target cargo, compensates the theoretical grasping pose based on the local imaging data, and generates the final grasping pose.

[0051] When the mobile platform 100 arrives at the target area (i.e., the area that the robotic arm can cover the target cargo), the target cargo pose provided by the global vision sensor 300 may be deviated due to measurement errors, platform positioning errors, or minor displacements of the cargo during transportation. This step activates the end-effector vision sensor 220 (e.g., a high-precision 3D camera or structured light camera, located at the end of the robotic arm and moving with it) to perform close-range local imaging of the target cargo, acquiring high-precision local imaging data to finely compensate for the theoretical grasping pose, ensuring the final grasping accuracy.

[0052] like Figure 5 As shown, in an optional embodiment, step S40 includes the following sub-steps: S401. When the mobile platform reaches the target area, the control end vision sensor performs local imaging of the target cargo.

[0053] After the controller detects that the mobile platform 100 has reached the target area, it sends an imaging command to the end vision sensor 220. The end vision sensor 220 performs a close-range local scan of the target cargo and collects high-precision three-dimensional point cloud data of the target cargo surface as local imaging data.

[0054] S402. Using the theoretically captured pose location as the center, extract the local region of interest data from the local imaging data.

[0055] The local imaging data contains a high-precision point cloud of the target cargo and its surrounding environment, using the theoretical grasping pose position point P calculated in step S204. grasp Centered on a target cargo and within a preset radius (e.g., 1.5 times the size of the target cargo), point cloud data within this radius is extracted from the local imaging data to form a Local Region of Interest (ROI). The ROI dataset contains information about the actual surface shape of the target cargo near its theoretical grasping pose.

[0056] S403. Perform local plane fitting on the local region of interest data to extract the actual surface normal vector and actual gripping surface geometric features of the target cargo at the theoretical gripping pose.

[0057] Perform local plane fitting on the extracted ROI data, for example, by fitting a plane equation using the least squares method, and obtain the normal vector of that plane as the surface normal vector of the target cargo at the actual gripping location. Simultaneously, calculate the centroid location P of all points in the ROI data. centroid As the geometric center of the actual gripping surface.

[0058] S404. Calculate the rotation compensation amount of the gripping posture based on the deviation angle between the actual surface normal vector and the preset ideal gripping normal vector, and calculate the translation compensation amount of the gripping position based on the centroid offset of the actual gripping surface geometric features.

[0059] Preset ideal grasping normal vector This refers to the theoretical surface normal vector of the target cargo under ideal conditions (i.e., without error), whose direction is the same as the unit vector of the grasping direction determined in step S205. Conversely, the end effector should approach the target cargo surface perpendicularly. (Actual surface normal vector) Due to measurement errors or cargo orientation deviations, there is a certain deviation from the ideal grasping normal vector.

[0060] Specifically, the calculation process for the rotation compensation and translation compensation in this step is as follows: First, calculate the actual surface normal vector. With the ideal grasping normal vector The angle between them is taken as the normal vector deviation angle θ: ; In the above formula, the dot product of two unit vectors is equal to the cosine of their included angle. The included angle can be obtained by using the inverse cosine function. The normal vector deviation angle θ reflects the degree of deviation between the actual posture and the theoretical posture of the target cargo.

[0061] Secondly, based on the actual surface normal vector With the ideal grasping normal vector The cross product determines the unit vector in the direction of the rotation axis. : ; In this formula, the cross product of two vectors yields a vector perpendicular to both vectors. The direction of this vector is the axis of rotation required to rotate the ideal normal vector to the direction of the actual normal vector. Normalizing the cross product result yields a unit vector. If the actual normal vector is in the same direction as the ideal normal vector (i.e., there is no deviation), then the cross product is zero, and no rotation compensation is needed.

[0062] Furthermore, based on the unit vector of the rotation axis direction Based on the deviation angle θ of the normal vector, a rotation matrix R is constructed according to the Rodriguez rotation formula, which serves as the rotation compensation amount for the grasping posture: ; in, It is the identity matrix. Unit vector in the direction of the rotation axis The antisymmetric matrix has the following form: ; k x k y k z They are vectors The components on the three coordinate axes (X-axis, Y-axis, Z-axis) in a three-dimensional coordinate system, antisymmetric matrices are used to transform cross product operations into matrix multiplication.

[0063] The formula for calculating the rotation matrix R above is the standard form of the Rodrigues rotation formula, used to construct rotations around a unit vector. The rotation matrix for rotation angle θ: any rotation can be represented as a rotation around an axis by a certain angle. The corresponding rotation matrix can be constructed using the rotation axis direction vector and the rotation angle.

[0064] Finally, the centroid position P of the actual gripping surface geometry is calculated. centroid The position point P of the theoretically captured pose grasp Spatial offset vector between As compensation for the translation of the grab position: .

[0065] The above formula can be used to calculate the spatial offset of the actual cargo surface center relative to the theoretical gripping position, that is, the magnitude and direction of the translation compensation required for the theoretical gripping position.

[0066] S405. The rotation compensation and translation compensation are superimposed on the theoretical grasping pose to generate the final grasping pose.

[0067] The rotation matrix R (rotation compensation amount) and translation vector calculated in step S404 are used to calculate the rotation matrix R (rotation compensation amount) and translation vector. (Translation compensation) acts on the homogeneous transformation matrix of the theoretical grasping pose. The homogeneous transformation matrix of the final grasping pose is obtained. : ; Based on the above formula, the final grasp pose is obtained by first performing attitude rotation compensation (rotating according to the rotation matrix R) and then performing position translation compensation (according to the translation vector). The actual grasping pose obtained after translation.

[0068] By activating a high-precision end-effector vision sensor for local imaging upon reaching the target area, the theoretical grasping pose is finely compensated, eliminating the impact of global vision sensor measurement errors, cumulative positioning errors of the mobile platform, and minute displacements of the goods during transportation on grasping accuracy. By calculating the deviation angle between the actual surface normal vector and the ideal grasping normal vector, the posture rotation compensation amount is constructed using the Rodrigues rotation formula, achieving posture deviation compensation in any direction. By calculating the offset vector between the centroid position of the actual grasping surface and the theoretical grasping point as the translation compensation amount, precise compensation for position deviation is achieved. The two-level vision strategy (global large-area perception and precise end-effector local imaging) balances perception range and positioning accuracy, enabling the robot to cover large loading and unloading areas while achieving high-precision final grasping.

[0069] S50: Control the robotic arm to grasp the target goods based on the final grasping posture.

[0070] The controller uses the final grasping pose generated in step S40 as the control target, driving each joint of the robot arm to move along the planned trajectory, so that the end effector 210 reaches the final grasping pose and performs the grasping action on the target cargo 400. The end effector 210 can be a gripper, suction cup, or other grasping device, and the appropriate end effector is selected according to the type of cargo.

[0071] In one embodiment, during the grabbing action in step S50, a security monitoring step is also included: First, the actual torque information of each joint of the robotic arm and the real-time image information collected by the end-effector are acquired in real time. For example, the servo drives of each joint of the robotic arm can output the joint torque value in real time, and the end-effector (or its matching 2D camera) simultaneously acquires real-time images of the target cargo.

[0072] Secondly, based on the actual torque information, it is determined whether the robotic arm has experienced an accidental collision. If an accidental collision is detected, the grasping action is immediately paused and an alarm is triggered. Specifically, if the torque value of a certain joint shows an abnormal peak exceeding the normal range within a short period of time, it is determined that the robotic arm has experienced an accidental collision with an obstacle in the environment. The grasping action is immediately paused and an alarm is triggered to avoid damage to the robotic arm or the goods.

[0073] Furthermore, after the end effector of the robotic arm contacts the target cargo, the relative slippage between the end effector and the target cargo is detected based on real-time image information. Specifically, by tracking feature points in the real-time images acquired by the end vision sensor (e.g., tracking the position changes of texture feature points on the cargo surface in the image), the relative displacement between the end effector and the cargo can be calculated.

[0074] Finally, if the relative slippage exceeds a preset threshold (e.g., 2mm), it is determined that the current clamping force is insufficient to stably grasp the goods. The clamping force of the end effector is increased and a second clamping is performed to ensure that the goods are reliably grasped before proceeding with subsequent handling and unloading actions.

[0075] Collision detection and protection are achieved through real-time torque monitoring, avoiding mechanical damage under abnormal working conditions. At the same time, closed-loop monitoring of gripping quality is achieved through visual slip detection. When slippage is detected, the clamping force is automatically increased and clamped again, which improves the reliability and safety of gripping and prevents goods from falling and causing damage or safety accidents during handling.

[0076] This invention identifies each cargo and its pose by acquiring 3D information of the loading / unloading area from a global vision sensor. Based on the pose of each cargo, a grasping decision is made to determine the target cargo and the theoretical grasping pose. As the mobile platform moves towards the target cargo, the relative pose of the target cargo with respect to the robotic arm is continuously updated, and the robotic arm's posture is adjusted in real time. When the mobile platform reaches the target area, local imaging compensation is performed using an end-effector vision sensor to generate the final grasping pose, and the robotic arm is controlled to grasp the target cargo. On the one hand, the grasping decision and theoretical grasping pose calculation for the target cargo are completed during the movement of the mobile platform, and the relative pose is continuously updated and the robotic arm's posture is adjusted in real time, ensuring effective grasping. The preparation and movement processes completely overlap in time, and the platform has already completed pre-alignment when it arrives at the target area, which greatly shortens the grasping cycle of a single cargo and significantly improves the overall efficiency of loading and unloading operations. On the other hand, the global vision sensor provides three-dimensional information of the loading and unloading area with a large field of view, completes cargo identification and global grasping planning, and the end vision sensor performs close-range local high-precision imaging after the platform arrives at the target area, and performs fine compensation for the theoretical grasping pose. The two levels of vision work together to ensure both the global field of view and planning capability of grasping decision-making and the high-precision positioning of the final grasping, overcoming the technical contradiction that it is difficult to balance between field of view and positioning accuracy with single-level vision.

[0077] Figure 6 A schematic diagram of a vision-guided robot loading and unloading control device provided in an embodiment of the present invention is shown below. Figure 6 As shown, the vision-guided robot loading and unloading control device of this invention includes: Cargo identification module 601 is used to acquire three-dimensional information of the loading and unloading area collected by the global vision sensor, and to identify each cargo and its pose based on the three-dimensional information; The theoretical grasping pose determination module 602 is used to make grasping decisions based on the poses of each of the goods, and to determine the target goods and the theoretical grasping pose. The robotic arm posture adjustment module 603 is used to update the relative posture of the target cargo with respect to the robotic arm based on the real-time posture of the mobile platform and the posture of the target cargo during the process of the mobile platform moving towards the target cargo, and to adjust the posture of the robotic arm based on the relative posture. The final grasping pose generation module 604 is used to control the end vision sensor to perform local imaging of the target cargo when the mobile platform reaches the target area, and to compensate the theoretical grasping pose based on the local imaging data to generate the final grasping pose. The gripping control module 605 is used to control the robotic arm to grip the target goods according to the final gripping pose.

[0078] The vision-guided robot loading and unloading control device described above can execute the vision-guided robot loading and unloading control method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0079] Figure 1 This is a schematic diagram of the robot according to an embodiment of the present invention, as shown below. Figure 1 As shown, the robot in this embodiment of the invention includes: Mobile platform 100; A robotic arm 200 mounted on the mobile platform 100 and an end vision sensor 220 mounted at the end of the robotic arm 200; Global vision sensor 300; One or more controllers; Storage device for storing one or more programs; The mobile platform 100, the global vision sensor 300, and the end vision sensor 220 all communicate with the controller; When the one or more programs are executed by the one or more controllers, the one or more controllers implement the vision-guided robot loading and unloading control method described in the embodiments of the present invention.

[0080] It should be noted that the descriptions of the device and robot embodiments are relatively simple because they are basically similar to the method embodiments. For relevant details, please refer to the descriptions of the method embodiments.

[0081] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause the bin robot to execute the vision-guided robot loading and unloading control method described in any embodiment of the present invention.

[0082] It is worth noting that the various units and modules included in the above-mentioned vision-guided robot loading and unloading control device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0083] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution device. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0084] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0085] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A vision-guided robot loading and unloading control method, characterized in that, Applied to a robot, the robot includes a mobile platform, a global vision sensor, a robotic arm mounted on the mobile platform, and an end-effector vision sensor mounted at the end of the robotic arm. The control method includes: The three-dimensional information of the loading and unloading area collected by the global vision sensor is obtained, and each cargo and its pose are identified based on the three-dimensional information. Based on the pose of each item, a grasping decision is made to determine the target item and the theoretical grasping pose; During the process of the mobile platform moving towards the target cargo, the relative pose of the target cargo with respect to the robotic arm is updated according to the real-time pose of the mobile platform and the pose of the target cargo, and the posture of the robotic arm is adjusted according to the relative pose. When the mobile platform reaches the target area, it controls the end vision sensor to perform local imaging of the target cargo, and compensates the theoretical grasping pose based on the local imaging data to generate the final grasping pose. The robotic arm is controlled to grasp the target cargo based on the final grasping pose.

2. The control method according to claim 1, characterized in that, Acquire the three-dimensional information of the loading and unloading area collected by the global vision sensor, and identify each cargo and its pose based on the three-dimensional information, including: The global vision sensor is controlled to perform a global scan of the loading and unloading area to acquire raw global point cloud data. The original global point cloud data is filtered, denoised, and segmented into planes to extract the effective point cloud data of the cargo stacking surface in the loading and unloading area. Based on the effective point cloud data, a point cloud clustering segmentation algorithm is used to independently segment each of the goods from the stacked point cloud, and a three-dimensional bounding box fitting is performed on each independently segmented point cloud cluster of goods to output the three-dimensional bounding box information of each of the goods and their pose in the global coordinate system.

3. The control method according to claim 2, characterized in that, Based on the pose of each item, a grasping decision is made to determine the target item and the theoretical grasping pose, including: Calculate the support and coverage relationship between the goods based on their poses, and determine the stacking level of each of the goods; Based on the stacking hierarchy, an initial fetching sequence is generated according to a priority rule from top to bottom and from outside to inside. During the movement of the mobile platform, it is detected in real time whether the first item in the initial grasping sequence meets the workspace constraints of the robotic arm. If it does, it is determined to be the target item. If it does not meet, it is postponed to the next item until an item that meets the workspace constraints is selected as the target item. After determining the target cargo, the position point of the theoretical grasping pose is calculated according to the following formula: ； in, The theoretically captured pose position point is located in three dimensions in the global coordinate system. Let be the geometric center coordinates of the three-dimensional bounding box of the target cargo, and d be the preset gripping offset distance. The grabbing direction unit vector, the grabbing direction unit vector Dynamically select based on the stacking level of the target goods: If the target cargo is not a top-level cargo, the grabbing direction unit vector is selected as a horizontal direction vector; if the target cargo is a top-level cargo, the grabbing direction unit vector is selected as a weighted composite vector of the vertical and horizontal direction vectors.

4. The control method according to claim 1, characterized in that, During the movement of the mobile platform toward the target cargo, the relative pose of the target cargo with respect to the robotic arm is updated based on the real-time pose of the mobile platform and the pose of the target cargo, including: Obtain the current platform pose in the global coordinate system; Based on the pose of the target cargo and the pose of the current platform, the relative pose is calculated according to the following formula: ； in, Let be the homogeneous transformation matrix corresponding to the relative pose of the target cargo with respect to the robotic arm. Let be the homogeneous transformation matrix corresponding to the current platform pose. Let be the homogeneous transformation matrix corresponding to the pose of the target cargo; Based on the moving speed of the mobile platform and the time rate of change of the relative pose, Kalman filtering is performed to predict the relative pose, generating a filtered smooth relative pose.

5. The control method according to claim 1, characterized in that, Adjusting the posture of the robotic arm based on the relative pose includes: Based on the relative pose and combined with the current angle information of each joint of the robotic arm, the desired joint angle combination of the robotic arm at the entrance of the target area is solved by inverse kinematics, so that the end effector of the robotic arm points to the target cargo. During the process of the mobile platform moving from its current position to the target area, the inverse kinematics solution is repeatedly executed at preset time intervals to generate continuous joint trajectories of each joint of the robotic arm, and the robotic arm is controlled to move according to the continuous joint trajectories so that the end effector can pre-align the target cargo before reaching the target area.

6. The control method according to claim 1, characterized in that, The theoretical grasping pose is compensated based on local imaging data to generate the final grasping pose, including: Using the theoretically captured pose location as the center, extract local region of interest data from the local imaging data; Local plane fitting is performed on the local region of interest data to extract the actual surface normal vector and actual gripping surface geometric features of the target cargo at the theoretical gripping pose; The rotation compensation amount of the gripping posture is calculated based on the deviation angle between the actual surface normal vector and the preset ideal gripping normal vector, and the translation compensation amount of the gripping position is calculated based on the centroid offset of the geometric features of the actual gripping surface. The rotation compensation and translation compensation are superimposed on the theoretical grasping pose to generate the final grasping pose.

7. The control method according to claim 6, characterized in that, Based on the deviation angle between the actual surface normal vector and the preset ideal gripping normal vector, the gripping posture rotation compensation is calculated, and based on the centroid offset of the actual gripping surface geometry, the gripping position translation compensation is calculated, including: Calculate the actual surface normal vector With the ideal grasping normal vector The angle between them is taken as the normal vector deviation angle θ: ； According to the actual surface normal vector With the ideal grasping normal vector The cross product determines the unit vector in the direction of the rotation axis. : ; According to the unit vector of the rotation axis direction The rotation matrix R is constructed using the normal vector deviation angle θ and the Rodriguez rotation formula, and is used as the rotation compensation amount for the grasping posture: ; in, It is the identity matrix. The unit vector in the direction of the rotation axis The antisymmetric matrix; Calculate the centroid position P of the actual gripping surface geometry. centroid The position point P of the theoretically grasped pose grasp Spatial offset vector between As the translation compensation amount for the grasping position: 。 8. The control method according to any one of claims 1-7, characterized in that, Also includes: The actual torque information of each joint of the robotic arm and the real-time image information collected by the end vision sensor are acquired in real time. Based on the actual torque information, determine whether the robotic arm has experienced an accidental collision. If an accidental collision is determined to have occurred, immediately pause the grasping process and trigger an alarm. After the end effector of the robotic arm contacts the target cargo, the relative slippage between the end effector and the target cargo is detected based on the real-time image information; If the relative slip exceeds a preset threshold, the clamping force of the end effector is increased and a second clamping is performed.

9. A vision-guided robot loading and unloading control device, characterized in that, Applied to robots, the robot includes a mobile platform, a global vision sensor, a robotic arm mounted on the mobile platform, and an end-effector vision sensor mounted at the end of the robotic arm, comprising: The cargo identification module is used to acquire the three-dimensional information of the loading and unloading area collected by the global vision sensor, and to identify each cargo and its pose based on the three-dimensional information. The theoretical grasping pose determination module is used to make grasping decisions based on the poses of each of the goods, and to determine the target goods and the theoretical grasping pose; The robotic arm posture adjustment module is used to update the relative posture of the target cargo with respect to the robotic arm based on the real-time posture of the mobile platform and the posture of the target cargo during the process of the mobile platform moving towards the target cargo, and to adjust the posture of the robotic arm based on the relative posture. The final grasping pose generation module is used to control the end vision sensor to perform local imaging of the target cargo when the mobile platform reaches the target area, and to compensate the theoretical grasping pose based on the local imaging data to generate the final grasping pose. The grasping control module is used to control the robotic arm to grasp the target goods according to the final grasping pose.

10. A robot, characterized in that, The robot includes: Mobile platform; A robotic arm mounted on the mobile platform and an end vision sensor mounted at the end of the robotic arm; Global vision sensor; One or more controllers; Storage device for storing one or more programs; The mobile platform, the global vision sensor, and the end vision sensor all communicate with the controller. When the one or more programs are executed by the one or more controllers, the one or more controllers implement the vision-guided robot loading and unloading control method as described in any one of claims 1-8.

Images