Robot disordered grabbing method and system based on 3D vision

Abstract

The invention discloses a robot disordered grabbing method and system based on 3D vision, and relates to the field of vision grabbing, and the method comprises the steps: configuring a 3D vision collection module to execute incoming material area continuous collection, executing definition of a robot base coordinate system, a camera coordinate system and an incoming material station coordinate system, and setting a tray in-place consistency constraint. Deterministic mapping of a clamp and a strategy is formed under multiple types of incoming materials, the grabbing success rate is improved, deformation and falling risks are reduced, collision safety is improved with minimum safety gap constraint and path cost selection in an unordered environment, and operation stability is maintained through action time accounting and beat conformity judgment on the beat side. And traceability is realized by a sorting accuracy target and a task log write-in mechanism, and finally, shutdown waiting and secondary faults are reduced through alarm and separate disposal rules in a static incoming material and structure separation risk scene.

Description

Technical Field

[0001] This invention relates to the field of visual grasping, and in particular to a method and system for unordered grasping of robots based on 3D vision. Background Technology

[0002] In warehousing and logistics operations, incoming materials are characterized by disordered stacking and mixed forms. Common targets include cardboard boxes, woven bags and burlap sacks, turnover boxes, and individual electrical fittings. When robots perform grasping and sorting in the in-frame, basket, and pallet areas, they usually rely on 3D vision to form point clouds and depth maps, and map the recognition results to the robot base coordinate system to realize a continuous operation process of target recognition, grasping pose generation, path planning, grasping confirmation, and classification and placement.

[0003] The existing disordered grasping scheme has the following shortcomings in engineering applications: First, the unified mapping between the camera coordinate system and the robot base coordinate system lacks the dual constraints of external parameter calibration residuals and pallet placement consistency. In AGV loading scenarios, pose drift is prone to occur, resulting in instability of candidate grasping areas and effective grasping poses. Second, in static material receiving and interlocking stacking scenarios, there is a lack of continuous alarm and manual handling closed loop after grasping status determination. The two unconnected parts of the fittings do not form separate handling rules, increasing the risk of grasping failure and falling. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a 3D vision-based method and system for unordered robotic grasping. This method aims to continuously acquire point cloud datasets and depth map datasets using 3D vision and bind them to acquisition timestamps. It establishes a robot base coordinate system, a camera coordinate system, and a material receiving station coordinate system. The system uses extrinsic parameter calibration and residual thresholds to lock the coordinate transformation matrix. Furthermore, it introduces pallet arrival consistency constraints in AGV loading scenarios to solidify relative relationships, thereby improving the uniformity and stability of target point cloud coordinates from the source. Based on target category labels, it triggers fixture selection and quick-change switching, generates a set of grasping poses, and calculates grasping pose scores. Combined with obstacle models and collision constraints in the unordered environment, it performs grasping confirmation. Under static material receiving conditions, it executes a closed loop of graspable state determination, continuous alarm for non-graspable conditions, and manual handling of temporary storage positions after offline processing. It also triggers specific alarms and separate handling rules for structural separation risk fittings.

[0005] Therefore, this application provides a method and system for unordered grasping of robots based on 3D vision, including the following steps: Step S100: Configure the 3D vision acquisition module to continuously acquire data in the material receiving area, define the robot base coordinate system, camera coordinate system, and material receiving station coordinate system, and set the pallet placement consistency constraint.

[0006] Step S200: Execute target object category determination, output target category label and candidate grasping area, set fixture selection strategy, bring out combined grasping strategy entry, execute fixture quick change device fixture switching, and set fixture control signal access parameters.

[0007] Step S300: Set the hardware parameter set, and execute the generation of the gripping pose set and the calculation of the gripping pose score value. Execute the screening of flat plane segments of small hardware and the confirmation of magnetic adsorption. Execute the generation of non-flat shape gripping boundary box and the confirmation of clamping in place. Execute the adsorption action and the bottom constraint verification of the gripper. Execute the triggering of the height detection probe and the verification of the gripper gripping height establishment signal.

[0008] Step S400: Configure the obstacle model and set the obstacle set, generate the grabbing path segment, set the disordered environment collision constraints and beat calculation parameters, execute the grabbing action, and complete the determination of the adsorption confirmation signal and the clamping in place confirmation signal.

[0009] Step S500: Read the target category label, output the placement strategy based on the target category label, execute the cycle control, set the cycle compliance judgment, set the sorting accuracy target, and write it into the task log for traceability.

[0010] Step S600: Set up a static incoming material trigger and graspable status determination mechanism, execute a closed loop of continuous alarm for non-grabable material and offline manual processing temporary storage position, and configure special alarm and separate handling rules for structural separation risk fittings.

[0011] In some specific embodiments, step S100 specifically includes: Step S100.1: Configure the 3D vision acquisition module to continuously acquire data in the material receiving area, and define the robot base coordinate system, camera coordinate system, and material receiving station coordinate system.

[0012] Step S100.2: Set the tray placement consistency constraint.

[0013] The AGV places the pallet into the bracket. The AGV's repeatability is set to ±10mm, and the AGV's repeatability angle deviation is set to ±1°. The bracket is equipped with a guard and a guide wheel. The guard and the guide wheel perform limit fitting correction on the outer edge of the pallet and are used to ensure the consistency of the pallet's positioning after it is placed in the bracket.

[0014] The incoming material station is set with pallet arrival determination logic, which includes edge contact status determination and guide wheel contact status determination. The pose error threshold of the pallet arrival determination logic is set to translation ±10mm and yaw angle ±1°. When the pallet arrival determination logic is valid, the incoming material station locks the relative relationship between the incoming material station coordinate system and the robot base coordinate system, and maps the point cloud dataset corresponding to the pallet incoming material area to generate the target point cloud dataset.

[0015] In some specific embodiments, step S200 specifically includes: Step S200.1: Perform target object category determination, output target category label and candidate grasping area, set fixture selection strategy, and bring up the combined grasping strategy entry.

[0016] Step S200.2: Perform fixture switching using the quick-change fixture device and set the fixture control signal input parameters.

[0017] The robot execution module is equipped with a quick-change fixture device, which includes quick-change master and slave trays. The quick-change master and slave trays support a load of 350kg. The robot execution module sets the fixture control signal access parameters, which include the input / output point capacity, the servo control group capacity, and the air passage capacity. The input / output point capacity is set to 219 points, the servo control group capacity is set to 4 groups, and the air passage capacity is set to 40 channels.

[0018] The robot execution module triggers the quick-change fixture device to perform the fixture switching process based on the target category label. After the fixture switching process is completed, it outputs a fixture ready status signal.

[0019] In some specific embodiments, step S300 specifically includes: Step S300.1: Set the set of fitting parameters, and execute the generation of the gripping pose set and the calculation of the gripping pose score. Execute the screening of flat plane segments of small fittings and the confirmation of magnetic adsorption. Execute the generation of non-flat shape gripping boundary box and the confirmation of clamping in place.

[0020] Step S300.2: Perform the adsorption action and check the bottom constraint of the gripper, and perform the height detection probe trigger and the hook gripping height establishment signal verification.

[0021] In some specific embodiments, step S400 specifically includes: Step S400.1: Configure the obstacle model, set the obstacle set, generate the grab path segment, and set the disordered environment collision constraints and beat calculation parameters.

[0022] Step S400.2: Execute the gripping action and complete the determination of the adsorption confirmation signal and the clamping in place confirmation signal.

[0023] The robot execution module drives the gripper to perform the approach, downward probe, and gripping actions according to the grasping execution queue. When the target category label is cardboard, the robot execution module drives the cardboard suction cup gripper to perform vacuum adsorption. The cardboard suction cup gripper is equipped with sponge suction cups or an array of evenly distributed multi-layer suction cups. The cardboard suction cup gripper is equipped with a one-way valve structure to isolate local air leakage. The robot execution module reads the output of the vacuum pressure sensor and sets the vacuum adsorption confirmation threshold to -55kPa. When the vacuum pressure sensor output meets -55kPa and remains stable for 120ms, the robot execution module determines that the vacuum adsorption confirmation is successful and drives the lifting and retraction actions.

[0024] If vacuum adsorption confirmation, magnetic adsorption confirmation, or clamping confirmation is not established, the robot execution module terminates the lifting segment and performs the retraction segment retreat action. The robot execution module records and marks the corresponding candidate grasping area as a grasping failure state and triggers the recalculation of the grasping path segment.

[0025] In some specific embodiments, step S500 specifically includes: Step S500.1: Read the target category label, output the placement strategy based on the target category label, execute the beat control, and set the beat compliance judgment.

[0026] Step S500.2: Set the sorting accuracy target and write it into the task log for traceability.

[0027] In some specific embodiments, step S600 specifically includes: Step S600.1: Set up a static material arrival trigger and graspable status determination mechanism, and execute a closed loop of continuous alarm for non-grabable and manual processing of temporary storage position after offline.

[0028] Step S600.2: Configure special alarm and separate handling rules for structural separation risk fittings.

[0029] The robot execution module establishes a set of structural separation risk fittings, which is limited to NX-2 and NUT-2. When the target category label is determined to be a single power fitting and the fitting parameter set matches the set of structural separation risk fittings, the robot execution module performs a two-part unconnected state determination on the candidate target object point cloud cluster. The robot execution module performs secondary segmentation on the candidate target object point cloud cluster and generates a first sub-cluster and a second sub-cluster. The robot execution module calculates the minimum connection distance between the first sub-cluster and the second sub-cluster.

[0030] When the two parts are not connected, the robot execution module outputs an alarm signal to the control cabinet and locks the task number. The human-machine interface outputs a prompt text and writes the separate storage and separate grasping rules: the separate storage and separate handling rules place the corresponding parts of the first sub-cluster and the corresponding parts of the second sub-cluster into two fixed partition positions in the offline manual processing temporary storage position. The two fixed partition positions are fixed with position vectors in the robot base coordinate system. The separate grasping and separate handling rules generate candidate grasping area records for the corresponding parts of the first sub-cluster and the corresponding parts of the second sub-cluster respectively and write them into the grasping execution queue. The time interval threshold between two grasping executions is set to 2 seconds to reduce the risk of grasping failure and falling.

[0031] A 3D vision-based robot unordered grasping system includes the following modules: The modeling and calibration module is used to configure the 3D vision acquisition module to continuously acquire data from the incoming material area, define the robot base coordinate system, camera coordinate system, and incoming material station coordinate system, and set consistency constraints for pallet placement.

[0032] The classification and selection module is used to determine the category of the target object, output the target category label and candidate grasping area, set the fixture selection strategy, bring out the combined grasping strategy entry, execute the fixture quick change device fixture switching, and set the fixture control signal access parameters.

[0033] The pose adaptation module is used to set the set of fitting parameters, and to perform the generation of the grasping pose set and the calculation of the grasping pose score value. It also performs the screening of flat plane segments of small fittings and the confirmation of magnetic adsorption, the generation of non-flat shape gripping boundary box and the confirmation of clamping in place, the adsorption action and the bottom constraint verification of the gripper, and the triggering of the height detection probe and the verification of the gripper gripping height establishment signal.

[0034] The collision planning and execution module is used to configure the obstacle model, set the obstacle set, generate the grabbing path segment, set the collision constraints and cycle calculation parameters of the disordered environment, execute the grabbing action, and complete the determination of the adsorption confirmation signal and the clamping in place confirmation signal.

[0035] The classification beat module is used to read the target category label, output the placement strategy based on the target category label, execute beat control, set the beat compliance judgment, set the sorting accuracy target, and write to the task log for traceability.

[0036] The static alarm module is used to set the static incoming material trigger and graspable status determination mechanism, execute the closed loop of continuous alarm for non-grabable material and temporary storage position for offline manual processing, and configure special alarm and separate handling rules for structural separation risk fittings.

[0037] In summary, this application provides a 3D vision-based robot disordered grasping method and system that forms a deterministic mapping between grippers and strategies under multiple types of incoming materials, improving the grasping success rate and reducing deformation and fall risks. In disordered environments, it enhances collision safety through minimum safety gap constraints and path cost selection. On the cycle time side, it maintains operational stability through motion time calculation and cycle time compliance determination, and achieves traceability through sorting accuracy targets and task log writing mechanisms. Finally, in static incoming material and structural separation risk scenarios, it reduces downtime and secondary failures through alarm and separate handling rules. Attached Figure Description

[0038] Figure 1 This is an overall flowchart of a robot unordered grasping method based on 3D vision provided in an embodiment of this application.

[0039] Figure 2 This is an overall framework diagram of a robot disordered grasping system based on 3D vision provided in an embodiment of this application. Detailed Implementation

[0040] Please refer to Figure 1 The diagram illustrates a flow of one embodiment of a 3D vision-based robot unordered grasping method and system according to the present disclosure.

[0041] like Figure 1 As shown, a method and system for unordered grasping by a robot based on 3D vision includes the following steps: Step S100: Configure the 3D vision acquisition module to continuously acquire data in the material receiving area, define the robot base coordinate system, camera coordinate system, and material receiving station coordinate system, and set the pallet placement consistency constraint.

[0042] Step S200: Execute target object category determination, output target category label and candidate grasping area, set fixture selection strategy, bring out combined grasping strategy entry, execute fixture quick change device fixture switching, and set fixture control signal access parameters.

[0043] Step S300: Set the hardware parameter set, and execute the generation of the gripping pose set and the calculation of the gripping pose score value. Execute the screening of flat plane segments of small hardware and the confirmation of magnetic adsorption. Execute the generation of non-flat shape gripping boundary box and the confirmation of clamping in place. Execute the adsorption action and the bottom constraint verification of the gripper. Execute the triggering of the height detection probe and the verification of the gripper gripping height establishment signal.

[0044] Step S400: Configure the obstacle model and set the obstacle set, generate the grabbing path segment, set the disordered environment collision constraints and beat calculation parameters, execute the grabbing action, and complete the determination of the adsorption confirmation signal and the clamping in place confirmation signal.

[0045] Step S500: Read the target category label, output the placement strategy based on the target category label, execute the cycle control, set the cycle compliance judgment, set the sorting accuracy target, and write it into the task log for traceability.

[0046] Step S600: Set up a static incoming material trigger and graspable status determination mechanism, execute a closed loop of continuous alarm for non-grabable material and offline manual processing temporary storage position, and configure special alarm and separate handling rules for structural separation risk fittings.

[0047] In some specific embodiments, step S100 specifically includes: Step S100.1: Configure the 3D vision acquisition module to continuously acquire data in the material receiving area, and define the robot base coordinate system, camera coordinate system, and material receiving station coordinate system.

[0048] The 3D vision acquisition module is equipped with a Mech-Mind camera. The Mech-Mind camera performs continuous acquisition of the material receiving area within the frame, the material receiving area in the basket, and the material receiving area in the pallet. The continuous acquisition outputs point cloud datasets and depth map datasets.

[0049] The 3D vision acquisition module is set to a continuous acquisition frequency of no less than 10Hz. The 3D vision acquisition module generates an acquisition sequence number for each frame of point cloud dataset and each frame of depth map dataset, and generates an acquisition timestamp for each frame of point cloud dataset and each frame of depth map dataset. The acquisition timestamp resolution is set to 1ms. The acquisition sequence number and acquisition timestamp are written into the point cloud dataset entry and the depth map dataset entry, and the point cloud dataset entry and the depth map dataset entry form a corresponding binding relationship.

[0050] The 3D vision acquisition module sets the point cloud sliding buffer capacity to 300 sets. The point cloud sliding buffer stores 300 sets of point cloud dataset entries and depth map dataset entries, which are used for subsequent verification of the effectiveness of camera robot extrinsic parameter calibration and stabilization mapping of target point cloud datasets.

[0051] The robot execution module establishes the robot base coordinate system and fixes the origin and axial direction of the robot base coordinate system. The Mech-Mind 3D camera mounting bracket establishes the camera coordinate system and fixes the origin and axial direction of the camera coordinate system. The material receiving station establishes the material receiving station coordinate system and fixes the origin and axial direction of the material receiving station coordinate system.

[0052] The robot execution module performs camera-robot extrinsic parameter calibration, and the output coordinate transformation matrix is ​​used to map the point cloud dataset in the camera coordinate system to the robot base coordinate system and generate the target point cloud dataset.

[0053] The robot execution module sets extrinsic parameter calibration residual thresholds. The position residual threshold is set to 2.0 mm, and the attitude residual threshold is set to 0.2°. When the position and attitude residuals meet the extrinsic parameter calibration residual thresholds, the robot execution module locks the coordinate transformation matrix and maintains the coordinate uniformity of the target point cloud dataset. When the position and attitude residuals exceed the extrinsic parameter calibration residual thresholds, the robot execution module keeps the coordinate transformation matrix unlocked and triggers extrinsic parameter calibration recalculation.

[0054] Step S100.2: Set the tray placement consistency constraint.

[0055] The AGV places the pallet into the bracket. The AGV's repeatability is set to ±10mm, and the AGV's repeatability angle deviation is set to ±1°. The bracket is equipped with a guard and a guide wheel. The guard and the guide wheel perform limit fitting correction on the outer edge of the pallet and are used to ensure the consistency of the pallet's positioning after it is placed in the bracket.

[0056] The incoming material station is set with pallet arrival determination logic, which includes edge contact status determination and guide wheel contact status determination. The pose error threshold of the pallet arrival determination logic is set to translation ±10mm and yaw angle ±1°. When the pallet arrival determination logic is valid, the incoming material station locks the relative relationship between the incoming material station coordinate system and the robot base coordinate system, and maps the point cloud dataset corresponding to the pallet incoming material area to generate the target point cloud dataset.

[0057] When the pallet arrival determination logic is not established, the material receiving station maintains the unlocked state of the material receiving station coordinate system. The 3D vision acquisition module continuously outputs point cloud datasets and depth map datasets and refreshes the point cloud sliding buffer until the pallet arrival determination logic is established and the target point cloud dataset is locked.

[0058] In some specific embodiments, step S200 specifically includes: Step S200.1: Perform target object category determination, output target category label and candidate grasping area, set fixture selection strategy, and bring up the combined grasping strategy entry.

[0059] The robot execution module reads the target point cloud dataset and performs target point cloud segmentation operation within the effective working area of ​​the material receiving station. The target point cloud segmentation operation outputs a target point cloud segmentation cluster set, and each target point cloud segmentation cluster in the target point cloud segmentation cluster set corresponds to a candidate target object point cloud cluster.

[0060] The robot execution module calculates the 3D circumscribed cuboid size parameters for the candidate target object point cloud cluster. The 3D circumscribed cuboid size parameters include length, width, and height. The robot execution module calculates the number of points in the plane and the total number of points in the point cloud for the candidate target object point cloud cluster. Based on the 3D circumscribed cuboid size parameters and the proportion of points in the plane, the robot execution module performs target category label determination. The target category labels are limited to cardboard boxes, woven bags and burlap sacks, turnover boxes, and single power fittings.

[0061] The robot execution module sets the carton size threshold to a maximum size of 600×500×500mm and a minimum size of 240×220×150mm. The robot execution module also sets the carton shape deformation threshold to 5mm. When the carton size threshold and the carton shape deformation threshold are met, the target category label is determined to be a carton.

[0062] The robot execution module sets the baseline size of woven bags and burlap sacks to 600×400×300mm, and sets the size deviation threshold of woven bags and burlap sacks to ±30mm. When the size deviation threshold of woven bags and burlap sacks is met, the target category label is determined to be woven bags and burlap sacks.

[0063] The robot execution module sets the turnover box size threshold to 600×400×400mm. When the turnover box size threshold is met, the target category label is determined to be a turnover box. When the cardboard box size threshold, woven bag and burlap bag reference size do not meet the turnover box size threshold, the target category label is determined to be a single power fitting.

[0064] The robot execution module generates candidate grasping regions based on target category labels. Each candidate grasping region consists of the geometric center point of the candidate target object's point cloud cluster, the set of surface normal vectors, and the grasping approach direction. These regions are then written into a candidate grasping region record for fixture selection and switching. Specifically: In the formula: It is the fixture matching score. It is the plane proportion weighting coefficient. It is a size matching weighting coefficient. It is a material compatibility weighting coefficient. It is the number of points in the plane of the point cloud cluster of the candidate target object. It is the total number of point clouds in the point cloud cluster of the candidate target object. It is a dimension vector consisting of length, width, and height. It is the nominal dimension vector of the fixture. It is the size difference attenuation coefficient and Set to 20mm. This indicates the material compatibility indicator; when a single power fitting meets the conditions for magnetic adsorption. Take 1, when a single power fitting does not meet the magnetic adsorption condition. Take 0.

[0065] The robot execution module establishes a fixture selection strategy, which establishes a correspondence between target category labels and fixture names. Fixture names include cardboard suction cup fixtures, sack fixtures, turnover box fixtures, and special hardware fixtures. When the target category label is cardboard, the robot execution module selects the cardboard suction cup fixture. The cardboard suction cup fixture is adapted to the cardboard box gripping weight threshold of 50kg and the cardboard box size threshold.

[0066] When the target category label is woven bags and burlap sacks, the robot execution module selects a burlap sack-specific gripper. The burlap sack-specific gripper is adapted to the woven bags and burlap sacks with a gripping weight threshold of 50kg and the deviation threshold between the woven bags and burlap sacks' reference dimensions and external dimensions. When the target category label is turnover boxes, the robot execution module selects a turnover box-specific gripper. The turnover box-specific gripper is adapted to the turnover box size threshold of 600×400×400mm and the gripping weight threshold of 50kg.

[0067] When the target category label is a single power fitting, the robot execution module selects a dedicated single-piece fixture. This fixture employs a combined gripping strategy of an 80mm magnetic chuck and pneumatic parallel grippers. The 80mm magnetic chuck is used for flat surface adhesion of a single power fitting, while the pneumatic parallel grippers are used for non-flat surface clamping. The robot execution module then determines the gripper based on the fixture matching score. Filter the execution pose of the combined crawling strategy entry point within the candidate crawling region record.

[0068] Step S200.2: Perform fixture switching using the quick-change fixture device and set the fixture control signal input parameters.

[0069] The robot execution module is equipped with a quick-change fixture device, which includes quick-change master and slave trays. The quick-change master and slave trays support a load of 350kg. The robot execution module sets the fixture control signal access parameters, which include the input / output point capacity, the servo control group capacity, and the air passage capacity. The input / output point capacity is set to 219 points, the servo control group capacity is set to 4 groups, and the air passage capacity is set to 40 channels.

[0070] The robot execution module triggers the quick-change fixture device to perform the fixture switching process based on the target category label. After the fixture switching process is completed, it outputs a fixture ready status signal.

[0071] In some specific embodiments, step S300 specifically includes: Step S300.1: Set the set of fitting parameters, and execute the generation of the gripping pose set and the calculation of the gripping pose score. Execute the screening of flat plane segments of small fittings and the confirmation of magnetic adsorption. Execute the generation of non-flat shape gripping boundary box and the confirmation of clamping in place.

[0072] When the fixture is ready signal is established, the robot execution module reads the candidate gripping area record with the target category label of a single power fitting. The robot execution module establishes a fitting parameter set, which includes the fitting size vector, fitting weight parameter, and fitting material parameter. The upper limit of the fitting weight parameter is set to 50kg, and the fitting material parameter is used to determine the magnetic adsorption conditions of the 80 magnetic chuck.

[0073] The robot execution module generates a set of grasping poses within the candidate grasping region record. The set of grasping poses includes a grasping position vector and a grasping posture matrix. The grasping proximity vector of the grasping posture matrix is ​​taken from the set of surface normal vectors of the candidate grasping region record.

[0074] The robot execution module calculates a grasping pose score for the grasping pose set and writes the grasping pose score into an entry in the grasping pose set, specifically: In the formula: It captures the pose score. It is the weighting coefficient for fixture matching score. These are the stability weighting coefficients for plane fitting. It is the safety clearance weighting coefficient. It is the fixture matching score. It captures the fitting residuals of the flat plane segment corresponding to the pose. It is the threshold for the fitting residual of a flat plane segment and Set to 2.0. It is the minimum safe clearance corresponding to the grasping pose. It is the minimum safety clearance calibration threshold and The robot execution module is set to 8.0, with a threshold of 0.75 for the grasping pose score. Grasping poses with a score of 0.75 are retained as valid grasping poses, while grasping poses with a score of less than 0.75 are removed from the grasping pose set.

[0075] The robot execution module sets a set of small fittings, which includes QP7, QP10, W-7B, WS-10, U-7, Z-7 and PH-10. When the target category label is a single power fitting and the fitting parameter set matches the set of small fittings, the robot execution module enables the 80 magnetic chuck adsorption mode.

[0076] The robot execution module performs flat plane segment screening within the candidate target object point cloud cluster. The flat plane segment screening adopts a plane fitting distance threshold, which is set to 2.0mm. The surface normal vector corresponding to the flat plane segment is used as the grasping approach vector. The robot execution module sets the magnetic adsorption position as the geometric center point of the flat plane segment.

[0077] The robot execution module sets the magnetic adsorption approach distance threshold to 30mm. The robot execution module drives the special metal fixture to complete the approach action along the grasping approach vector. The 80 magnetic chuck performs the adsorption action and outputs a magnetic adsorption confirmation signal. The magnetic adsorption confirmation signal is kept stable for 120ms. When the magnetic adsorption confirmation signal meets the 120ms requirement, the robot execution module locks the valid grasping posture corresponding to the magnetic adsorption condition and enters S400.

[0078] When the target category label is a single power fitting and the magnetic adsorption confirmation signal does not meet 120ms, the robot execution module enables the pneumatic parallel gripper clamping mode. The robot execution module generates a gripping bounding box within the candidate target object point cloud cluster. The gripping bounding box includes the gripping center position vector and the gripping opening direction vector.

[0079] The robot execution module sets the target value of the gripper opening, which is composed of the boundary width of the gripping boundary box plus a safety margin. The safety margin is set to 6mm. The robot execution module sets the target value of the clamping force to 60N to 110N, with the higher value taken as the weight parameter of the hardware increases.

[0080] The pneumatic parallel gripper performs the gripping action and outputs a clamping confirmation signal. The clamping confirmation signal is held stably for 150ms. When the clamping confirmation signal meets the 150ms requirement, the robot execution module locks the effective gripping posture corresponding to the pneumatic parallel gripper gripping condition.

[0081] Step S300.2: Perform the adsorption action and check the bottom constraint of the gripper, and perform the height detection probe trigger and the hook gripping height establishment signal verification.

[0082] When the target category label is cardboard box, the robot execution module reads the name of the cardboard box suction cup clamp. The robot execution module sets the maximum size of the cardboard box to 600×500×500mm and the minimum size of the cardboard box to 240×220×150mm. The robot execution module sets the threshold for the deformation of the cardboard box shape to 5mm. When the deformation of the cardboard box shape meets 5mm and the size of the cardboard box meets the maximum and minimum sizes, the lifting mechanism of the cardboard box suction cup clamp performs a descent action to complete the suction action. After the lifting mechanism performs an upward action, the gripper of the cardboard box suction cup clamp performs a closing action to form a bottom constraint.

[0083] When the target category label is woven bag and burlap sack, the robot execution module reads the name of the burlap sack special gripper. The robot execution module sets the dimensional deviation threshold of woven bag and burlap sack to ±30mm and the gripping weight threshold to 50kg. When the dimensional deviation of woven bag and burlap sack meets ±30mm and the gripping weight meets 50kg, the burlap sack special gripper performs the gripping action to complete the suction action. The burlap sack special gripper maintains the gripping state and simultaneously performs the bottom constraint closing action.

[0084] When the target category label is turnover box, the robot execution module reads the name of the turnover box special fixture. The robot execution module sets the turnover box size threshold to 600×400×400mm. When the turnover box size threshold is met, the robot execution module drives the turnover box special fixture to move above the turnover box candidate gripping area.

[0085] The height detection probe of the special gripper for the turnover box performs a downward movement. After the height detection probe triggers the gripping height establishment signal, the hooks on both sides of the special gripper for the turnover box perform a gripping action and output a hook gripping position confirmation signal. The stable holding time of the hook gripping position confirmation signal is set to 150ms. When the hook gripping position confirmation signal meets 150ms, the robot execution module locks the effective gripping posture of the turnover box.

[0086] In some specific embodiments, step S400 specifically includes: Step S400.1: Configure the obstacle model, set the obstacle set, generate the grab path segment, and set the disordered environment collision constraints and beat calculation parameters.

[0087] After the fixture is ready, the robot execution module reads the target point cloud dataset and the target category label. The robot execution module then builds an obstacle model in the robot base coordinate system. The obstacle model is represented by voxels, and the voxel resolution is set to 5mm.

[0088] The robot execution module constructs an obstacle set, which includes the material frame boundary, the basket boundary, the bracket stop edge, and identified but not yet grasped target objects. The material frame boundary, the basket boundary, and the bracket stop edge are written into the obstacle set by the structural parameters of the material receiving station. The identified but not yet grasped target objects are written into the obstacle set by the candidate target object point cloud clusters that have not entered the grasp confirmation state from the target point cloud segmentation cluster set.

[0089] The robot execution module performs an expansion process on the obstacle set. The expansion radius is set to 10mm. The expansion process outputs a safe obstacle set, which is used to determine the collision constraints of the grasping path segment.

[0090] The robot execution module reads the valid grasping pose and generates a grasping path segment based on the grasping approach vector corresponding to the valid grasping pose. The grasping path segment includes an approach segment, a downward probe segment, a gripping action segment, a lifting segment, and a withdrawal segment. The pose at the end of the approach segment maintains pose continuity with the pose at the beginning of the downward probe segment, the pose at the end of the downward probe segment maintains pose continuity with the pose at the beginning of the gripping action segment, the pose at the beginning of the lifting segment maintains pose continuity with the pose at the end of the gripping action segment, and the pose at the beginning of the withdrawal segment maintains pose continuity with the pose at the end of the lifting segment.

[0091] The robot execution module sets collision constraints in the disordered environment. The collision constraints adopt a minimum safe gap threshold, which is set to 10mm. The robot execution module discretely samples 100 path points in each grasping path segment. The robot execution module calculates the minimum distance between the robot body envelope and the set of safe obstacles corresponding to each path point. Grasping path segments with a minimum distance of less than 10mm are determined to be unexecutable grasping path segments and trigger the recalculation of grasping path segments.

[0092] The robot execution module sets the robot's linear speed to 2 m / s for cycle time calculation. The robot execution module also sets the robot speed percentage to 90%. The robot execution module calculates the robot's linear speed based on this percentage and uses it for calculating the time interval for grasping the path. Specifically: In the formula: It is the value of the crawling path. It is the number of discrete sampling points captured on the path segment and Take 100, It is the gap penalty coefficient and Set to 1.0. It is the first The minimum distance from the robot's body envelope to the set of safety obstacles at each sampling point. It is a distance-stable term and Set to 1mm It is the joint smoothness coefficient and Set to 0.02. It is the first The joint displacement increment vector at each sampling point.

[0093] The robot execution module is based on the cost of grasping the path. Select the crawl path segment with the lowest crawl path cost from the candidate crawl path segment set as the crawl path segment to be executed, and write the executed crawl path segment into the crawl execution queue.

[0094] Step S400.2: Execute the gripping action and complete the determination of the adsorption confirmation signal and the clamping in place confirmation signal.

[0095] The robot execution module drives the gripper to perform the approach, downward probe, and gripping actions according to the grasping execution queue. When the target category label is cardboard, the robot execution module drives the cardboard suction cup gripper to perform vacuum adsorption. The cardboard suction cup gripper is equipped with sponge suction cups or an array of evenly distributed multi-layer suction cups. The cardboard suction cup gripper is equipped with a one-way valve structure to isolate local air leakage. The robot execution module reads the output of the vacuum pressure sensor and sets the vacuum adsorption confirmation threshold to -55kPa. When the vacuum pressure sensor output meets -55kPa and remains stable for 120ms, the robot execution module determines that the vacuum adsorption confirmation is successful and drives the lifting and retraction actions.

[0096] When the target category label is a single power fitting and the 80 magnetic chuck is in the enabled state, the robot execution module drives the 80 magnetic chuck to perform the adsorption action and reads the magnetic adsorption confirmation signal. The magnetic adsorption confirmation signal is kept stable for 120ms. When the magnetic adsorption confirmation signal meets 120ms, the robot execution module determines that the magnetic adsorption confirmation is successful and drives the lifting section and the evacuation section.

[0097] When the target category label is a single power fitting and the pneumatic parallel gripper is in the enabled state, the robot execution module drives the pneumatic parallel gripper to perform a clamping action and reads the clamping confirmation signal. The clamping confirmation signal is kept stable for 150ms. When the clamping confirmation signal meets 150ms, the robot execution module determines that the clamping confirmation is successful and drives the lifting section and the retraction section.

[0098] If vacuum adsorption confirmation, magnetic adsorption confirmation, or clamping confirmation is not established, the robot execution module terminates the lifting segment and performs the retraction segment retreat action. The robot execution module records and marks the corresponding candidate grasping area as a grasping failure state and triggers the recalculation of the grasping path segment.

[0099] In some specific embodiments, step S500 specifically includes: Step S500.1: Read the target category label, output the placement strategy based on the target category label, execute the beat control, and set the beat compliance judgment.

[0100] Once the fixture is ready and the gripping is confirmed, the robot execution module reads the target category label and establishes a placement strategy. This strategy binds the target category label to the unloading station and outputs the placement pose. When the target category label is a cardboard box, the placement strategy outputs the cardboard box unloading station pose; when the target category label is a woven bag or burlap sack, the placement strategy outputs the woven bag or burlap sack unloading station pose; and when the target category label is a turnover box, the placement strategy outputs the turnover box unloading station pose.

[0101] When the target category label is a single power fitting, the robot execution module reads the fitting parameter set and generates a fitting model label. The fitting model label is limited to QP7, QP10, W-7B, WS-10, U-7, Z-7 and PH-10. The placement strategy binds the fitting model label to the specified category temporary storage position and outputs the placement pose of the specified category temporary storage position.

[0102] The robot execution module drives the end pose of the evacuation section to transition to the placement approach pose according to the placement pose. The placement approach pose and the placement pose maintain the same direction constraint for the grasping approach vector. The placement approach distance threshold is set to 40mm.

[0103] After the robot execution module completes the placement action, it outputs a placement completion status signal. The stable holding time of the placement completion status signal is set to 120ms.

[0104] The robot execution module establishes motion time calculation parameters, which include the suction time of the lifting mechanism and the execution time of the gripping path segment. The suction time of the lifting mechanism of the cardboard suction cup clamp is set to 2.5s, the suction time of the lifting mechanism of the burlap sack clamp is set to 2.5s, the overall motion time of the cardboard suction cup clamp is set to 14s, the overall motion time of the burlap sack clamp is set to 14s, and the overall motion time of the turnover box clamp is set to 12s, while meeting the cycle target of 5 pieces / min.

[0105] The robot execution module sets an upper limit time threshold for the cycle time. The upper limit time threshold is 14 seconds for the cardboard suction cup gripper and the sack gripper, and 12 seconds for the turnover box gripper. The robot execution module calculates the actual action time for each gripping execution queue and outputs the cycle time compliance. Specifically: In the formula: It's about rhythmic accuracy. It is the upper limit of the time threshold for the beat. It is the actual action time, which is obtained by adding up the time of the approach segment, the downward probe segment, the clamping action segment, the lifting segment, the withdrawal segment, and the placement action segment.

[0106] The robot's execution module is set to a cycle time compliance threshold of 0. When the condition is met (0), the robot execution module maintains the robot speed percentage at 90%. If the condition is not met (0), the robot execution module will reduce the robot speed percentage to 80% and trigger a recalculation of the grasping path segment. The recalculation of the grasping path segment will prioritize the grasping path cost. The crawling path segment has a low number of path points and a path count of 100.

[0107] Step S500.2: Set the sorting accuracy target and write it into the task log for traceability.

[0108] The robot's execution module is set to achieve a sorting accuracy target of 99%. It generates a task log entry for each gripping operation, which includes the target category label, hardware model label, valid gripping pose, fixture name, fixture number, and gripping pose score. Fixture matching score Value of crawling paths The system includes signals for vacuum adsorption confirmation, magnetic adsorption confirmation, clamping confirmation, placement completion, and execution result marking. The fixture number is assigned and fixed as a unique number by the quick-change device for carton suction cup fixtures, sack fixtures, turnover box fixtures, and special hardware fixtures.

[0109] After the placement completion status signal is established, the robot execution module writes the task log entry to the task log. The task log writing period is set to no more than 200ms. When the execution result is marked as successful, the robot execution module enters the next target point cloud dataset processing flow. When the execution result is marked as failed, the robot execution module marks the candidate grasping area record as grasping failure status and triggers the grasping pose set update.

[0110] In some specific embodiments, step S600 specifically includes: Step S600.1: Set up a static material arrival trigger and graspable status determination mechanism, and execute a closed loop of continuous alarm for non-grabable and manual processing of temporary storage position after offline.

[0111] After the 3D vision acquisition module outputs the point cloud dataset, the robot execution module extracts the inter-frame point cloud difference value. The inter-frame point cloud difference value is used to characterize the overall change range of the point cloud dataset entries between two adjacent frames. The inter-frame point cloud difference value is defined as follows: In the formula: It is the inter-frame point cloud difference value. It is the first Number of points in the frame point cloud dataset It is the first Frame number Point cloud points, It is the first Frame and The corresponding point cloud points after index alignment.

[0112] The robot execution module sets the static material arrival judgment threshold to 1.0mm and the static material arrival duration frame threshold to 30 frames. When the point cloud difference value between frames meets 1.0mm for 30 consecutive frames, the robot execution module determines that the static material arrival state is established.

[0113] After the static material arrival state is established, the robot execution module performs a graspable state determination on the target point cloud segmentation cluster set. The graspable state determination simultaneously satisfies the exposure surface establishment determination, the disordered environment collision constraint establishment determination, and the gripper entry establishment determination: The robot execution module reads the set of surface normal vectors and the grasping approach direction recorded in the candidate grasping area. The robot execution module sets the angle threshold between the surface normal vector and the grasping approach vector to 20°. The robot execution module sets the exposure surface point ratio threshold to 0.35. When the exposure surface point ratio meets 0.35, the robot execution module determines that the exposure surface is established.

[0114] The robot execution module reads the set of safety obstacles and verifies the minimum safety gap of 100 path points discretely sampled from the grasping path segment. The minimum safety gap threshold is maintained at 10mm. When the minimum safety gap of 100 path points meets 10mm, the robot execution module determines that the collision constraint of the disordered environment is valid.

[0115] The robot execution module reads the gripping bounding box for the pneumatic parallel gripper gripping condition. The robot execution module sets the gripper entry clearance distance threshold to 8mm. When the gripper entry clearance distance meets 8mm, the robot execution module determines that the gripper has entered the clearance.

[0116] When at least one candidate target object point cloud cluster in the target point cloud segmentation cluster set meets the graspable state determination, the robot execution module maintains the grasping execution queue. When none of the target point cloud segmentation cluster sets meet the graspable state determination, the robot execution module enters a closed loop of continuous alarm for non-graspable and offline manual processing temporary storage.

[0117] The robot execution module introduces a non-gripable continuous frame counter. The non-gripable continuous frame counter increments frame by frame in the static material receiving state as the non-gripable state occurs. The robot execution module sets the non-gripable continuous frame threshold to 50 frames. When the continuous sampling frequency is not less than 10Hz, 50 frames correspond to a duration of 5s.

[0118] When the ungraspable continuous frame counter reaches 50 frames, the robot execution module locks the task number and outputs an alarm signal to the control cabinet. The alarm signal includes an audible and visual alarm output and a human-machine interface prompt text output. The human-machine interface prompt text output includes the task number and the offline manual processing temporary storage instructions.

[0119] The robot execution module introduces a temporary storage location for manual processing after the robot is offline. The position vector of the temporary storage location for manual processing after the robot is offline is fixed in the robot base coordinate system. The disposal instructions for the temporary storage location for manual processing after the robot is offline include manual untangling action, manual re-swing action, and re-identification trigger conditions. The robot execution module sets the re-identification trigger condition to be that the tray is in place and the difference value of the point cloud between frames meets 1.0mm for 10 consecutive frames. After the re-identification trigger condition is met, the robot execution module releases the task number locking state and clears the counter for the number of frames that cannot be grasped continuously. The robot execution module rereads the target point cloud dataset and refreshes the target point cloud segmentation cluster set and candidate grasping area record.

[0120] Step S600.2: Configure special alarm and separate handling rules for structural separation risk fittings.

[0121] The robot execution module establishes a set of structural separation risk fittings, which is limited to NX-2 and NUT-2. When the target category label is determined to be a single power fitting and the fitting parameter set matches the set of structural separation risk fittings, the robot execution module performs a two-part unconnected state determination on the candidate target object point cloud cluster. The robot execution module performs secondary segmentation on the candidate target object point cloud cluster and generates a first sub-cluster and a second sub-cluster. The robot execution module calculates the minimum connection distance between the first sub-cluster and the second sub-cluster, which is defined as: In the formula: It is the minimum connection distance. It is the set of points in the first sub-cluster. It is the set of points in the second sub-cluster.

[0122] The robot execution module sets the minimum connection distance threshold to 6mm. When the minimum connection distance meets 6mm, the robot execution module determines that the two parts are not connected.

[0123] When the two parts are not connected, the robot execution module outputs an alarm signal to the control cabinet and locks the task number. The human-machine interface outputs a prompt text and writes the separate storage and separate grasping rules: the separate storage and separate handling rules place the corresponding parts of the first sub-cluster and the corresponding parts of the second sub-cluster into two fixed partition positions in the offline manual processing temporary storage position. The two fixed partition positions are fixed with position vectors in the robot base coordinate system. The separate grasping and separate handling rules generate candidate grasping area records for the corresponding parts of the first sub-cluster and the corresponding parts of the second sub-cluster respectively and write them into the grasping execution queue. The time interval threshold between two grasping executions is set to 2 seconds to reduce the risk of grasping failure and falling.

[0124] like Figure 2 As shown, a robot unordered grasping system based on 3D vision includes the following modules: The modeling and calibration module is used to configure the 3D vision acquisition module to continuously acquire data from the incoming material area, define the robot base coordinate system, camera coordinate system, and incoming material station coordinate system, and set consistency constraints for pallet placement.

[0125] The classification and selection module is used to determine the category of the target object, output the target category label and candidate grasping area, set the fixture selection strategy, bring out the combined grasping strategy entry, execute the fixture quick change device fixture switching, and set the fixture control signal access parameters.

[0126] The pose adaptation module is used to set the set of fitting parameters, and to perform the generation of the grasping pose set and the calculation of the grasping pose score value. It also performs the screening of flat plane segments of small fittings and the confirmation of magnetic adsorption, the generation of non-flat shape gripping boundary box and the confirmation of clamping in place, the adsorption action and the bottom constraint verification of the gripper, and the triggering of the height detection probe and the verification of the gripper gripping height establishment signal.

[0127] The collision planning and execution module is used to configure the obstacle model, set the obstacle set, generate the grabbing path segment, set the collision constraints and cycle calculation parameters of the disordered environment, execute the grabbing action, and complete the determination of the adsorption confirmation signal and the clamping in place confirmation signal.

[0128] The classification beat module is used to read the target category label, output the placement strategy based on the target category label, execute beat control, set the beat compliance judgment, set the sorting accuracy target, and write to the task log for traceability.

[0129] The static alarm module is used to set the static incoming material trigger and graspable status determination mechanism, execute the closed loop of continuous alarm for non-grabable material and temporary storage position for offline manual processing, and configure special alarm and separate handling rules for structural separation risk fittings.

[0130] In practical applications, the Mech-Mind 3D camera installed at the material receiving station forms a 3D vision acquisition module. It continuously acquires point cloud datasets and depth map datasets for the material receiving areas within the frame, the material basket receiving area, and the pallet receiving area. The continuous acquisition frequency is set to no less than 10Hz, the acquisition timestamp resolution is set to 1ms, and the point cloud sliding buffer capacity is set to 300 sets. The robot execution module solidifies the robot base coordinate system, camera coordinate system, and material receiving station coordinate system and performs camera-robot extrinsic parameter calibration. The position residual threshold is set to 2.0mm and the attitude residual threshold is set to 0.2°. The AGV places the pallet into the bracket. The repeatability positioning accuracy is set to ±10mm and the angle deviation is set to ±1°. After the pallet arrival judgment logic is valid, the relative relationship of the material receiving station coordinate system is locked and the target point cloud dataset is generated.

[0131] The robot execution module reads the target point cloud dataset, performs target point cloud segmentation within the effective working area of ​​the material receiving station, and generates a target point cloud segmentation cluster set. For each candidate target object point cloud cluster, it calculates the circumscribed cuboid size parameters and the proportion of points in the plane. The target category labels are limited to cardboard boxes, woven bags and burlap sacks, turnover boxes, and single electrical fittings. The cardboard box size threshold is set to a maximum of 600×500×500mm and a minimum of 240×220×150mm, with a shape deformation threshold of 5mm. The woven bag and burlap sack base size is set to 600×400×300mm, with a shape size deviation threshold of ±30mm. The turnover box size threshold is set to 600×400×400mm. The candidate grasping area consists of the geometric center point, the surface normal vector set, and the grasping approach direction, and is written into the candidate grasping area record.

[0132] The quick-change fixture device supports a load of 350kg. The fixture control signal input parameter settings include 219 input / output points, 4 servo control groups, and 40 pneumatic channels. After fixture switching, a fixture ready status signal is output. A single power fitting establishes a fitting parameter set and limits the weight to 50kg. The gripping posture scoring threshold is set to 0.75 to filter valid gripping postures. Small fittings use an 80 magnetic chuck with a plane fitting distance threshold of 2.0mm and an approach distance threshold of 30mm. The confirmation signal is held stably for 120ms. If the magnetic confirmation is not established, a pneumatic parallel gripper is used with a safety margin of 6mm and a clamping force of 60N to 110N. The clamping confirmation signal is held stably for 150ms. After the carton is adsorbed, the gripper holds the bottom with a deformation threshold of 5mm. After the turnover box probe is triggered, the hook gripping confirmation signal is held stably for 150ms.

[0133] The robot execution module establishes a voxel-based obstacle model in the robot base coordinate system, with a voxel resolution of 5mm. The obstacle set includes the material frame boundary, basket boundary, bracket edge, and ungrabbed target object. A safety obstacle set is generated with an expansion radius of 10mm. Under effective grasping posture constraints, the module generates approach, downward, gripping, lifting, and withdrawal segments. The collision constraint sets a minimum safe clearance threshold of 10mm and discretely samples 100 path points. If the clearance is less than 10mm, a recalculation is triggered. The cycle time calculation uses a linear speed of 2m / s and a speed percentage of 90%. The vacuum adsorption confirmation threshold is set to -55kPa and maintained stably for 120ms. The magnetic adsorption confirmation signal is maintained stably for 120ms, and the clamping confirmation signal is maintained stably for 150ms. If any confirmation is not established, the module performs withdrawal and refreshes the grasping queue.

[0134] After the grab is confirmed, the robot execution module reads the target category label, establishes a placement strategy, and outputs the placement pose. Cardboard boxes are bound to the cardboard box discharge station, woven bags and burlap sacks are bound to the woven bag and burlap sack discharge stations, and turnover boxes are bound to the turnover box discharge station. Individual power fittings generate fitting model labels and are bound to a designated category temporary storage location. The placement proximity threshold is set to 40mm, the placement completion status signal is maintained for 120ms, the action time calculation parameter is set to 2.5s for the lifting mechanism suction time, the upper limit threshold for the cycle time of the cardboard box suction cup clamp and the burlap sack clamp is 14s, the upper limit threshold for the cycle time of the turnover box clamp is 12s and meets the requirement of 5 pieces / min, the cycle time compliance threshold is set to 0, when it meets the requirement, the speed percentage is maintained at 90%, when it does not meet the requirement, it is reduced to 80% and the path is recalculated, the sorting accuracy target is set to 99%, and the task log writing cycle is set to no more than 200ms and records the clamp number and execution result.

[0135] The inter-frame point cloud difference value is used for static material arrival triggering. The threshold is set to 1.0mm and the continuous frame count threshold is 30 frames. If 1.0mm is met for 30 consecutive frames, the static material arrival state is determined. Under the static material arrival state, the graspable state is determined: the included angle threshold is 20°, the exposed surface point ratio threshold is 0.35, the minimum safety gap threshold is 10mm, and the gripper entry clearance distance threshold is 8mm. If none of these conditions are met, the ungraspable counter is incremented and the threshold is set to 50 frames. When the counter meets the threshold, the task number is locked and an alarm is output to the control cabinet, and a manual handling temporary storage location disposal guide is given. The triggering conditions are re-identified and the pallet arrival determination logic is set to be met for 10 consecutive frames. The structural separation risk fittings are limited to NX-2 and NUT-2. When the minimum connection distance threshold is set to 6mm, a special alarm is triggered and partition storage is executed. The threshold for the interval between two grasps is set to 2s.

Claims

1. A method for unordered grasping of a robot based on 3D vision, characterized in that, Includes the following steps: S100: Configure a 3D vision acquisition module to continuously acquire data from the incoming material area, define the robot base coordinate system, camera coordinate system, and incoming material station coordinate system, and set pallet placement consistency constraints. S200: Perform target object category determination, output target category label and candidate grasping area, set fixture selection strategy, bring out combined grasping strategy entry, execute fixture quick change device fixture switching, and set fixture control signal access parameters. S300, set the hardware parameter set, and execute the generation of the gripping posture set and the calculation of the gripping posture score value, execute the screening of flat plane segments of small hardware and the confirmation of magnetic adsorption, execute the generation of non-flat shape gripping boundary box and the confirmation of clamping in place, execute the adsorption action and the bottom constraint verification of the gripper, and execute the height detection probe trigger and the hook gripping height establishment signal verification. S400: Configure obstacle model and set obstacle set, generate grabbing path segment, set disordered environment collision constraints and beat calculation parameters, execute grabbing action, and complete adsorption confirmation signal and clamping in place confirmation signal determination. S500: Reads target category labels, outputs placement strategies based on target category labels, executes cycle control, sets cycle compliance judgment, sets sorting accuracy targets, and writes to the task log for traceability; S600: Set up static incoming material trigger and graspable status determination mechanism, execute continuous alarm for non-grabable material and closed loop for offline manual processing temporary storage position, and configure special alarm and separate handling rules for structural separation risk fittings.

2. The robot unordered grasping method based on 3D vision according to claim 1, characterized in that, S100 specifically includes: S100.1 Configure the 3D vision acquisition module to continuously acquire data in the material receiving area and define the robot base coordinate system, camera coordinate system, and material receiving station coordinate system. The 3D vision acquisition module is equipped with a Mech-Mind camera. The Mech-Mind camera performs continuous acquisition of the material receiving area within the frame, the material receiving area in the basket, and the material receiving area in the pallet. The continuous acquisition outputs point cloud datasets and depth map datasets. The 3D vision acquisition module is set to a continuous acquisition frequency of no less than 10Hz. The 3D vision acquisition module generates an acquisition sequence number for each frame of point cloud dataset and each frame of depth map dataset. The 3D vision acquisition module generates an acquisition timestamp for each frame of point cloud dataset and each frame of depth map dataset. The acquisition timestamp resolution is set to 1ms. The acquisition sequence number and acquisition timestamp are written into the point cloud dataset entry and the depth map dataset entry, and the point cloud dataset entry and the depth map dataset entry form a corresponding binding relationship. The robot execution module establishes the robot base coordinate system and fixes the origin position and axial direction of the robot base coordinate system. The Mech-Mind camera 3D camera mounting bracket establishes the camera coordinate system and fixes the origin position and axial direction of the camera coordinate system. The material receiving station establishes the material receiving station coordinate system and fixes the origin position and axial direction of the material receiving station coordinate system. The robot execution module performs camera-robot extrinsic parameter calibration. The camera-robot extrinsic parameter calibration outputs a coordinate transformation matrix. The coordinate transformation matrix is ​​used to map the point cloud dataset in the camera coordinate system to the robot base coordinate system and generate the target point cloud dataset. S100.2, Set pallet placement consistency constraints; The AGV places the pallet into the bracket. The AGV's repeatability is set to ±10mm, and the AGV's repeatability angle deviation is set to ±1°. The bracket is equipped with a guard and a guide wheel. The guard and the guide wheel perform limit fitting correction on the outer edge of the pallet and are used to ensure the consistency of the pallet's positioning after it is placed in the bracket. The incoming material station is set with pallet arrival determination logic, which includes edge contact status determination and guide wheel contact status determination. The pose error threshold of the pallet arrival determination logic is set to translation ±10mm and yaw angle ±1°. When the pallet arrival determination logic is valid, the incoming material station locks the relative relationship between the incoming material station coordinate system and the robot base coordinate system, and maps the point cloud dataset corresponding to the pallet incoming material area to generate the target point cloud dataset.

3. The robot unordered grasping method based on 3D vision according to claim 1, characterized in that, S200 specifically includes: S200.1 Execute target object category determination, output target category label and candidate grasping area, set fixture selection strategy, and bring up the combined grasping strategy entry point; The robot execution module reads the target point cloud dataset. The robot execution module performs the target point cloud segmentation operation within the effective working area of ​​the material receiving station. The target point cloud segmentation operation outputs a target point cloud segmentation cluster set. Each target point cloud segmentation cluster in the target point cloud segmentation cluster set corresponds to a candidate target object point cloud cluster. The robot execution module calculates the three-dimensional circumscribed cuboid size parameters for the candidate target object point cloud cluster. The three-dimensional circumscribed cuboid size parameters include length, width, and height. The robot execution module calculates the number of points in the plane and the total number of points in the point cloud for the candidate target object point cloud cluster. Based on the three-dimensional circumscribed cuboid size parameters and the proportion of points in the plane, the robot execution module performs target category label determination. The target category labels are limited to cardboard boxes, woven bags and burlap sacks, turnover boxes, and single power fittings. The robot execution module sets the carton size threshold to a maximum size of 600×500×500mm and a minimum size of 240×220×150mm. The robot execution module also sets the carton shape deformation threshold to 5mm. When the carton size threshold and the carton shape deformation threshold are met, the target category label is determined to be a carton. The robot execution module sets the baseline size of woven bags and burlap sacks to 600×400×300mm, and sets the size deviation threshold of woven bags and burlap sacks to ±30mm. When the size deviation threshold of woven bags and burlap sacks is met, the target category label is determined to be woven bags and burlap sacks. The robot execution module sets the turnover box size threshold to 600×400×400mm. When the turnover box size threshold is met, the target category label is determined to be a turnover box. When the cardboard box size threshold, the woven bag and burlap bag reference size do not meet the turnover box size threshold, the target category label is determined to be a single power fitting. The robot execution module generates candidate grasping regions based on target category labels. The candidate grasping region consists of the geometric center point of the candidate target object point cloud cluster, the set of surface normal vectors, and the grasping approach direction. The candidate grasping region record is written to the candidate grasping region record for fixture selection and fixture switching. The robot execution module establishes a fixture selection strategy, which establishes a correspondence between target category labels and fixture names. Fixture names include cardboard suction cup fixtures, sack fixtures, turnover box fixtures, and special hardware fixtures. When the target category label is cardboard, the robot execution module selects the cardboard suction cup fixture. The cardboard suction cup fixture is adapted to the cardboard box gripping weight threshold of 50kg and the cardboard box size threshold. When the target category label is a single power fitting, the robot execution module selects a dedicated single-piece fixture. This fixture employs a combined gripping strategy of an 80mm magnetic chuck and pneumatic parallel grippers. The 80mm magnetic chuck is used for flat surface adhesion of a single power fitting, while the pneumatic parallel grippers are used for non-flat surface clamping. The robot execution module then determines the gripper based on the fixture matching score. Filter the execution pose of the combined crawling strategy entry point within the candidate crawling area record; S200.2 Execute the quick-change fixture switching device to switch fixtures and set the fixture control signal input parameters; The robot execution module is equipped with a quick-change fixture device, which includes quick-change master and slave trays. The quick-change master and slave trays support a load of 350kg. The robot execution module sets the fixture control signal access parameters, which include the input / output point capacity, the servo control group capacity, and the air passage capacity. The input / output point capacity is set to 219 points, the servo control group capacity is set to 4 groups, and the air passage capacity is set to 40 channels. The robot execution module triggers the quick-change fixture device to perform the fixture switching process based on the target category label. After the fixture switching process is completed, it outputs a fixture ready status signal.

4. The robot unordered grasping method based on 3D vision according to claim 1, characterized in that, The S300 specifically includes: S300.1 Set the set of fitting parameters, and execute the generation of the gripping pose set and the calculation of the gripping pose score. Execute the screening of flat plane segments of small fittings and the confirmation of magnetic adsorption. Execute the generation of non-flat shape gripping boundary box and the confirmation of clamping in place. When the fixture is ready signal is established, the robot execution module reads the candidate grasping area record with the target category label of a single power fitting. The robot execution module establishes a fitting parameter set, which includes the fitting size vector, fitting weight parameter, and fitting material parameter. The upper limit of the fitting weight parameter is set to 50kg, and the fitting material parameter is used to determine the magnetic adsorption conditions of the 80 magnetic chuck. The robot execution module generates a set of grasping poses within the candidate grasping region record. The set of grasping poses includes a grasping position vector and a grasping posture matrix. The grasping proximity vector of the grasping posture matrix is ​​taken from the set of surface normal vectors of the candidate grasping region record. The robot execution module calculates the grasping pose score for the grasping pose set and writes the grasping pose score into the grasping pose set entry; The robot execution module sets a set of small fittings, which includes QP7, QP10, W-7B, WS-10, U-7, Z-7 and PH-10. When the target category label is a single power fitting and the fitting parameter set matches the set of small fittings, the robot execution module enables the 80 magnetic chuck adsorption mode. When the target category label is a single power fitting and the magnetic adsorption confirmation signal does not meet 120ms, the robot execution module activates the pneumatic parallel gripper clamping mode. The robot execution module generates a gripping bounding box within the candidate target object point cloud cluster. The gripping bounding box includes the gripping center position vector and the gripping opening direction vector. The robot execution module sets the target value of the gripper opening, which is composed of the boundary width of the gripping boundary box plus a safety margin. The safety margin is set to 6mm. The robot execution module sets the target value of the clamping force to 60N to 110N, with the higher value taken as the weight parameter of the hardware increases. S300.2, Perform adsorption action and claw bottom constraint verification, and perform height detection probe trigger and claw gripping height establishment signal verification; When the target category label is cardboard box, the robot execution module reads the name of the cardboard box suction cup clamp. The robot execution module sets the maximum size of the cardboard box to 600×500×500mm and the minimum size of the cardboard box to 240×220×150mm. The robot execution module sets the threshold for the deformation of the cardboard box shape to 5mm. When the deformation of the cardboard box shape meets 5mm and the size of the cardboard box meets the maximum and minimum size of the cardboard box, the lifting mechanism of the cardboard box suction cup clamp performs a descent action to complete the suction action. After the lifting mechanism performs an upward action, the gripper of the cardboard box suction cup clamp performs a closing action to form a bottom constraint. When the target category label is woven bag and burlap sack, the robot execution module reads the name of the burlap sack special gripper. The robot execution module sets the dimensional deviation threshold of woven bag and burlap sack to ±30mm and the gripping weight threshold to 50kg. When the dimensional deviation of woven bag and burlap sack meets ±30mm and the gripping weight meets 50kg, the burlap sack special gripper performs the gripping action to complete the suction action. The burlap sack special gripper maintains the gripping state and simultaneously performs the bottom constraint closing action. When the target category label is turnover box, the robot execution module reads the name of the turnover box special fixture. The robot execution module sets the turnover box size threshold to 600×400×400mm. When the turnover box size threshold is met, the robot execution module drives the turnover box special fixture to move above the turnover box candidate gripping area.

5. The robot unordered grasping method based on 3D vision according to claim 1, characterized in that, The S400 specifically includes: S400.1 Configure the obstacle model, set the obstacle set, generate the grab path segment, and set the disordered environment collision constraints and beat calculation parameters; After the fixture is ready signal is established, the robot execution module reads the target point cloud dataset and the target category label. The robot execution module establishes an obstacle model in the robot base coordinate system. The obstacle model is expressed using voxel occupancy, and the voxel resolution is set to 5mm. The robot execution module constructs an obstacle set, which includes the material frame boundary, the basket boundary, the bracket stop edge, and identified but not grasped target objects. The material frame boundary, the basket boundary, and the bracket stop edge are written into the obstacle set by the structural parameters of the material receiving station. The identified but not grasped target objects are written into the obstacle set by the candidate target object point cloud clusters that have not entered the grasp confirmation state in the target point cloud segmentation cluster set. The robot execution module reads the valid grasping pose and generates a grasping path segment with the grasping approach vector corresponding to the valid grasping pose as a constraint. The grasping path segment includes an approach segment, a downward probe segment, a gripping action segment, a lifting segment, and a withdrawal segment. The pose at the end of the approach segment is continuous with the pose at the beginning of the downward probe segment, the pose at the end of the downward probe segment is continuous with the pose at the beginning of the gripping action segment, the pose at the beginning of the lifting segment is continuous with the pose at the end of the gripping action segment, and the pose at the beginning of the withdrawal segment is continuous with the pose at the end of the lifting segment. The robot execution module sets collision constraints in the disordered environment. The collision constraints adopt a minimum safe gap threshold, which is set to 10mm. The robot execution module discretely samples 100 path points in each grasping path segment. The robot execution module calculates the minimum distance between the robot body envelope and the set of safe obstacles corresponding to each path point. Grasping path segments with a minimum distance of less than 10mm are determined to be unexecutable grasping path segments and trigger the recalculation of grasping path segments. The robot execution module sets the robot's linear speed to 2m / s for cycle time calculation. The robot execution module sets the robot speed percentage to 90%. The robot execution module converts the robot's linear speed according to the robot speed percentage and uses it for calculating the grasping path segment time. The robot execution module is based on the cost of grasping the path. Select the crawl path segment with the lowest crawl path cost from the candidate crawl path segment set as the crawl path segment to be executed, and write the crawl path segment to be executed into the crawl execution queue. S400.2 Execute the gripping action and complete the determination of the adsorption confirmation signal and the clamping in place confirmation signal; The robot execution module drives the gripper to perform the approach, downward probe, and gripping actions according to the grasping execution queue. When the target category label is a cardboard box, the robot execution module drives the cardboard suction cup gripper to perform a vacuum adsorption action. The cardboard suction cup gripper is equipped with sponge suction cups or an array of evenly distributed multi-layer suction cups. The cardboard suction cup gripper is equipped with a one-way valve structure to isolate local air leakage. The robot execution module reads the output of the vacuum pressure sensor and sets the vacuum adsorption confirmation threshold to -55kPa. When the vacuum pressure sensor output meets -55kPa and remains stable for 120ms, the robot execution module determines that the vacuum adsorption confirmation is successful and drives the lifting and retraction actions. If vacuum adsorption confirmation, magnetic adsorption confirmation, or clamping confirmation is not established, the robot execution module terminates the lifting segment and performs the retraction segment retreat action. The robot execution module records and marks the corresponding candidate grasping area as a grasping failure state and triggers the recalculation of the grasping path segment.

6. The robot unordered grasping method based on 3D vision according to claim 1, characterized in that, The S500 specifically includes: S500.1 Read the target category label, output the placement strategy based on the target category label, execute the beat control, and set the beat compliance judgment; Once the fixture is ready and the gripping is confirmed, the robot execution module reads the target category label and establishes a placement strategy. The placement strategy binds the target category label to the unloading station and outputs the placement pose. When the target category label is a carton, the placement strategy outputs the placement pose of the carton unloading station. When the target category label is a woven bag or burlap sack, the placement strategy outputs the placement pose of the woven bag or burlap sack unloading station. When the target category label is a turnover box, the placement strategy outputs the placement pose of the turnover box unloading station. When the target category label is a single power fitting, the robot execution module reads the fitting parameter set and generates a fitting model label. The fitting model label is limited to QP7, QP10, W-7B, WS-10, U-7, Z-7 and PH-10. The placement strategy binds the fitting model label to the specified category temporary storage position and outputs the placement pose of the specified category temporary storage position. The robot execution module drives the end pose of the evacuation section to transition to the placement approach pose according to the placement pose. The placement approach pose and the placement pose maintain the same direction constraint of the grasping approach vector. The placement approach distance threshold is set to 40mm. The robot execution module establishes motion time calculation parameters, which include the suction time of the lifting mechanism and the execution time of the gripping path segment. The suction time of the lifting mechanism of the cardboard suction cup clamp is set to 2.5s, the suction time of the lifting mechanism of the burlap sack clamp is set to 2.5s, the overall motion time of the cardboard suction cup clamp is set to 14s, the overall motion time of the burlap sack clamp is set to 14s, and the overall motion time of the turnover box clamp is set to 12s, while meeting the cycle target of 5 pieces / min. The robot execution module sets the upper limit time threshold for the cycle time. The upper limit time threshold for the cycle time is 14 seconds for the suction cup gripper for cardboard boxes and the gripper for burlap sacks, and 12 seconds for the gripper for turnover boxes. The robot execution module calculates the actual action time for each grasping execution queue and outputs the cycle time compliance. S500.2 Set the sorting accuracy target and write it to the task log for traceability; The robot's execution module is set to achieve a sorting accuracy target of 99%. It generates a task log entry for each gripping operation, which includes the target category label, hardware model label, valid gripping pose, fixture name, fixture number, and gripping pose score. Fixture matching score Value of crawling paths The system includes signals for vacuum adsorption confirmation, magnetic adsorption confirmation, clamping confirmation, placement completion, and execution result marking. The fixture number is assigned and fixed as a unique number by the quick-change device for carton suction cup fixtures, sack fixtures, turnover box fixtures, and special hardware fixtures. After the placement completion status signal is established, the robot execution module writes the task log entry to the task log. The task log writing period is set to no more than 200ms. When the execution result is marked as successful, the robot execution module enters the next target point cloud dataset processing flow. When the execution result is marked as failed, the robot execution module marks the candidate grasping area record as grasping failure status and triggers the grasping pose set update.

7. The robot unordered grasping method based on 3D vision according to claim 1, characterized in that, The S600 specifically includes: S600.1, Set up a static material arrival trigger and graspable status determination mechanism, and execute a closed loop of continuous alarm for non-grabable and manual processing of temporary storage position after offline; After the 3D vision acquisition module outputs the point cloud dataset, the robot execution module extracts the inter-frame point cloud difference value, which is used to characterize the overall change range of the point cloud dataset entries between two adjacent frames. The robot execution module sets the static material arrival judgment threshold to 1.0mm and the static material arrival duration frame threshold to 30 frames. When the point cloud difference value between frames meets 1.0mm for 30 consecutive frames, the robot execution module determines that the static material arrival state is established. After the static material arrival state is established, the robot execution module performs a graspable state determination on the target point cloud segmentation cluster set. The graspable state determination simultaneously satisfies the exposure surface establishment determination, the disordered environment collision constraint establishment determination, and the gripper entry establishment determination: the robot execution module reads the set of surface normal vectors and the grasping approach direction recorded in the candidate grasping area. The robot execution module sets the angle threshold between the surface normal vector and the grasping approach vector to 20°. The robot execution module sets the exposure surface point ratio threshold to 0.

35. When the exposure surface point ratio meets 0.35, the robot execution module determines that the exposure surface is established. The robot execution module introduces an ungraspable continuous frame counter. The ungraspable continuous frame counter accumulates frame by frame in the static material receiving state as the ungraspable state occurs. The robot execution module sets the ungraspable continuous frame threshold to 50 frames. When the continuous sampling frequency is not less than 10Hz, 50 frames correspond to a duration of 5s. When the ungraspable continuous frame counter reaches 50 frames, the robot execution module locks the task number and outputs an alarm signal to the control cabinet. The alarm signal includes an audible and visual alarm output and a human-machine interface prompt text output. The human-machine interface prompt text output includes the task number and the offline manual processing temporary storage instructions. The robot execution module introduces a temporary storage position for manual processing after the offline process. The position vector of the temporary storage position for manual processing after the offline process is fixed in the coordinate system of the robot base. The disposal guide of the temporary storage position for manual processing after the offline process is written with manual untangling action, manual re-swing action and re-identification trigger condition. The robot execution module sets the re-identification trigger condition as the tray arrival judgment logic is true and the difference value of the point cloud between frames meets 1.0mm for 10 consecutive frames. After the re-identification trigger condition is true, the robot execution module releases the task number locking state and clears the counter for the number of frames that cannot be grasped continuously. The robot execution module rereads the target point cloud dataset and refreshes the target point cloud segmentation cluster set and candidate grasping area record. S600.2, Configure special alarm and separate handling rules for structural separation risk fittings; The robot execution module establishes a set of structural separation risk fittings, which is limited to NX-2 and NUT-2. When the target category label is determined to be a single power fitting and the fitting parameter set matches the set of structural separation risk fittings, the robot execution module performs a two-part unconnected state determination on the candidate target object point cloud cluster. The robot execution module performs secondary segmentation on the candidate target object point cloud cluster and generates a first sub-cluster and a second sub-cluster. The robot execution module calculates the minimum connection distance between the first sub-cluster and the second sub-cluster. When the two parts are not connected, the robot execution module outputs an alarm signal to the control cabinet and locks the task number. The human-machine interface outputs a prompt text and writes the separate storage and separate grasping rules: the separate storage and separate handling rules place the corresponding parts of the first sub-cluster and the corresponding parts of the second sub-cluster into two fixed partition positions in the offline manual processing temporary storage position. The two fixed partition positions are fixed with position vectors in the robot base coordinate system. The separate grasping and separate handling rules generate candidate grasping area records for the corresponding parts of the first sub-cluster and the corresponding parts of the second sub-cluster respectively and write them into the grasping execution queue. The time interval threshold between two grasping executions is set to 2 seconds to reduce the risk of grasping failure and falling.

8. A robot unordered grasping system based on 3D vision, executing a three-dimensional warehouse scheduling method based on a four-way shuttle as described in any one of claims 1-7, characterized in that, Includes the following modules: The modeling and calibration module is used to configure the 3D vision acquisition module to continuously acquire data from the incoming material area, define the robot base coordinate system, camera coordinate system, and incoming material station coordinate system, and set consistency constraints for pallet placement. The classification and selection module is used to perform target object category determination, output target category labels and candidate grasping areas, set fixture selection strategies, bring out the combined grasping strategy entry, execute fixture quick change device fixture switching, and set fixture control signal access parameters. The pose adaptation module is used to set the set of fitting parameters, and to perform the generation of the grasping pose set and the calculation of the grasping pose score value. It also performs the screening of flat plane segments of small fittings and the confirmation of magnetic adsorption, the generation of non-flat shape gripping boundary box and the confirmation of clamping in place, the adsorption action and the bottom constraint verification of the gripper, and the triggering of the height detection probe and the verification of the gripping height establishment signal of the hook. The collision planning and execution module is used to configure the obstacle model, set the obstacle set, generate the grabbing path segment, set the collision constraints and cycle calculation parameters of the disordered environment, execute the grabbing action, and complete the determination of the adsorption confirmation signal and the clamping in place confirmation signal. The classification beat module is used to read the target category label, output the placement strategy based on the target category label, execute beat control, set the beat compliance judgment, set the sorting accuracy target, and write to the task log for traceability; The static alarm module is used to set the static incoming material trigger and graspable status determination mechanism, execute the closed loop of continuous alarm for non-grabable material and temporary storage position for offline manual processing, and configure special alarm and separate handling rules for structural separation risk fittings.

Images