Grabbing and loading method based on material visual self-calibration and mechanical hand

Abstract

The application discloses a kind of based on material vision self-calibration's grab and installs method and manipulator, comprising: the image and state information of first station, second station and end tool are collected, detect image quality, identify equipment reference datum, material natural feature and install entry boundary, the deviation between camera coordinate system, manipulator coordinate system, station coordinate system and end tool coordinate system is estimated on-line, and according to calibration quality generation grab and install candidate point and continuous grab and install trajectory.Manipulator is completed by composite end tool clamping, adsorption or lifting, and according to vision, weight, material level or end state review result updates calibration parameter and grab point score.The application is suitable for powder, granule, bagged, barrelled or tray material between conveying, screening, cooling, batching, feeding and isolation station Automatic grab and install.

Description

Technical Field

[0001] This invention relates to the field of automated material handling and industrial robot vision control technology, and particularly to a material handling method and robotic arm based on material vision self-calibration. Background Technology

[0002] In pharmaceutical, fine chemical, food, and similar production scenarios, powders, granules, bagged, drummed, or palletized materials typically need to be transferred between conveying, screening, cooling, weighing, feeding, and isolation stations. Existing automated production lines mostly use robotic arms in conjunction with conveying equipment to perform grasping or stacking. Some systems use fixed cameras to identify the material's location, and then the robotic arm performs the picking and placing actions according to a preset trajectory. These methods can reduce some manual handling, but they usually rely on pre-calibrated results and relatively fixed station locations.

[0003] During continuous production, factors such as conveyor belt misalignment, equipment vibration, hopper replacement, changes in hopper opening position, opening and closing of isolation doors, contamination of the vision window, and changes in material packaging shape can all lead to deviations between camera coordinates, robot coordinates, end effector coordinates, and target loading inlet coordinates. If static calibration or single-shot vision positioning is still used, problems such as empty gripping, off-center gripping, unstable clamping, loading misalignment, collision with the loading inlet, or the need to stop the machine for recalibration are likely to occur.

[0004] Furthermore, the contours, postures, heights, and gripping areas of soft bags, drums, boxes, sieved materials, and cooled materials vary considerably. Simply generating gripping points based on the center point or outer contour makes it difficult to simultaneously ensure gripping stability and loading accuracy. Existing robotic arms typically lack closed-loop verification of the gripping state, loading results, and historical deviations after gripping, failing to feed back the current gripping error to the coordinate calibration and gripping point selection in the next cycle. Therefore, it is necessary to provide a material gripping method and robotic arm capable of online self-calibration by combining station reference benchmarks, material natural characteristics, loading inlet boundaries, and end-effector status. Summary of the Invention

[0005] To address the problems of existing material handling systems that rely on static calibration, lack adaptability to changes in material shape, disconnect between the gripping and loading processes, and lack of post-loading verification and feedback, this invention provides a gripping method and robotic arm based on material vision self-calibration. This method acquires images and status information of the first and second workstations and the robotic arm's end effector during each gripping cycle. Combined with equipment reference standards, material natural characteristics, and loading inlet boundaries, coordinate deviations are estimated online, enabling the robotic arm to correct gripping actions based on workstation drift, loading inlet changes, and material posture variations in the production environment.

[0006] One aspect of the present invention provides a material gripping and loading method based on visual self-calibration. The method includes acquiring images and status information of a first station, a second station, and a robotic arm end effector, and performing quality inspection on the images; when the quality inspection meets the gripping and loading conditions, identifying the equipment reference datum at the first station, the natural characteristics of the material to be gripped, and the loading entrance boundary at the second station; estimating the deviation increments between the camera coordinate system, the robotic arm base coordinate system, the first station coordinate system, the second station coordinate system, and the end effector coordinate system online based on the equipment reference datum, the material's natural characteristics, and the loading entrance boundary, and forming a calibration quality; generating candidate gripping points for the material based on the deviation increments and the calibration quality, scoring the candidate gripping points, and determining the current gripping point; planning a continuous trajectory for gripping, transferring, aligning, loading, and exiting based on the current gripping point, the loading entrance boundary, and the end effector status; controlling the robotic arm to execute gripping actions according to the continuous trajectory, and acquiring the end effector status during execution; acquiring verification information after loading, and updating the deviation increment, the historical evaluation of the candidate gripping points, and the action compensation amount for the next cycle based on the verification results.

[0007] Specifically, the first station can be the end of a conveyor line, the discharge end of a screening device, the discharge end of a cooler, a buffer pallet, a bag loading platform, a drummed material turnover area, a weighing and batching station, or a transfer station within an isolation chamber. The second station can be a hopper, a barrel, a reactor inlet, a mixer inlet, a crusher inlet, a weighing container, a carrier within an isolation chamber, a subsequent conveyor line, or a stacking pallet. The material to be handled can be soft bags, barrels, boxes, pallet loads, screened material, cooled material, or packaged items.

[0008] Specifically, image quality detection can include evaluating image sharpness, brightness uniformity, reference datum visibility, depth data integrity, and occlusion / contamination ratio. When the image quality meets normal loading conditions, the system enters the process of identifying the equipment reference datum, material natural features, and loading entrance boundary. When the image quality is within the acceptable range, the system reduces the robot arm speed and takes a second picture. When the image quality is below the acceptable range, the system pauses the loading action and triggers supplementary lighting adjustment, visual window cleaning, or anomaly alerts. Through this processing, the system does not directly generate loading actions based on low-quality images when there is dust, reflection, visual window contamination, or partial target occlusion.

[0009] Specifically, the equipment reference datum includes a first reference feature fixed at the first station and a second reference feature fixed at the second station. The first reference feature is used to determine the positional deviation of the station where the material to be loaded is located relative to the theoretical station, and the second reference feature is used to determine the loading inlet center, loading inlet posture, and the accessible area of ​​the loading inlet. The first and second reference features can be formed by coded markings, circular holes, straight edges, corner points, bosses, cleaning-resistant calibration blocks, pallet corners, conveyor belt sides, or hopper boundaries. The material's natural characteristics include the material outline, material top height, material edge direction, material surface flatness, and material obstruction area. The system associates the equipment reference datum with the material's natural characteristics, so that coordinate calibration does not rely solely on the offline calibration plate or the material's outer contour.

[0010] Furthermore, the system saves the initial coordinate transformation relationship and calculates the deviation increment ΔT within the current loading cycle. The deviation increment ΔT represents the changes in the current camera coordinates, robot arm base coordinates, first station coordinates, second station coordinates, and end-effector coordinates relative to the initial calibration state. The system minimizes the weighted distance between the current measured position of the reference benchmark and the corresponding theoretical position, and combines the deviation increment from the previous cycle for smoothing constraints. The calibration quality Q can be calculated according to Q=αc_m+βc_r+γc_d+δc_l-ηc_o, where c_m is the confidence level of material characteristics, c_r is the confidence level of reference benchmark identification, c_d is the completeness of depth data, c_l is the confidence level of loading entrance boundary, c_o is the proportion of occlusion and contamination, and α, β, γ, δ, and η are weighting coefficients. Automatic loading is executed when the calibration quality reaches the normal loading threshold; low-speed loading or secondary shooting is executed when it is in the verification range; and automatic loading is stopped when it is below the verification range.

[0011] Furthermore, the candidate grabbing points are generated from the stable region within the material outline, the reinforced region at the material edge, the flat region on the top of the material, and the historically successful grabbing region. Each candidate grabbing point is associated with its spatial location, grabbing posture, approach direction, clamping opening, adsorption position, lifting position, and loading posture. The candidate grabbing point score S(g_i) can be calculated as S(g_i)=w_1V_i+w_2A_i+w_3B_i+w_4H_i+w_5R_i-w_6C_i-w_7D_i-w_8P_i, where Vi_i is visibility, A_i is end-point reachability, B_i is boundary margin, H_i is high stability, R_i is historical success rate, C_i is collision risk, D_i is material deformation or slippage risk, P_i is penalty for loading inlet posture mismatch, and w_1 to w_8 are weighting coefficients. The system selects the candidate grab point that satisfies the safety constraints and has the best ranking as the current grab point, so that the grab point selection simultaneously conforms to grab stability and loading feasibility.

[0012] Furthermore, the continuous trajectory includes an approach segment, a gripping segment, a lifting segment, a transfer segment, a loading inlet alignment segment, a descent loading segment, a release segment, and an exit segment. The approach segment determines the approach angle based on the normal or edge direction of the current gripping point; the lifting segment determines the safe height based on the material height and surrounding obstacles; the loading inlet alignment segment corrects the robot's posture based on the loading inlet center, loading inlet posture, and accessible area of ​​the second station; the descent loading segment limits the descent depth based on the material size and end-effector shape; and the exit segment departs from the second station along a direction avoiding the loading inlet edge. When the deviation increment changes, the key trajectory nodes of the continuous trajectory are corrected synchronously with the deviation increment.

[0013] Furthermore, the robot arm simultaneously collects the end-effector status during gripping actions. The end-effector status includes clamping force F_c, suction negative pressure P_v, gripper stroke L_j, tool attitude angle θ_g, and slippage state s_g. After approaching the current gripping point, the robot arm first enters the contact state at a low speed, then closes the gripper and establishes suction negative pressure based on the material size. Subsequently, it determines whether the gripping is reliable by measuring the clamping force, suction negative pressure, and gripper stroke. When the clamping force is below the lower limit, the suction negative pressure is below the lower limit, the gripper stroke is inconsistent with the visually estimated material size, or the tool attitude angle changes abnormally, the system determines that the current gripping poses a risk of empty gripping, biased gripping, or slippage, and executes actions such as releasing, re-gripping, reducing speed, or switching to a backup gripping point.

[0014] Furthermore, the post-loading verification information includes the image of the second station loading entrance, the material loading position, the target area occupancy status, material level information, weight information, and end-effector residue status. When the verification result shows a deviation of the material relative to the target loading position, the error e=q_a-q_t between the actual loading position q_a and the theoretical loading position q_t is decomposed into visual positioning error, end-effector error, and material slippage error, and the historical evaluations of coordinate deviation compensation, end-effector compensation, and loading candidate points are updated accordingly. When the verification result shows that the material is not fully loaded, there is material hanging on the loading entrance edge, or there is end-effector residue, the system generates a secondary adjustment action or a re-grabbing action and writes the corresponding failure reason into the history record.

[0015] Another aspect of the present invention provides a material handling robot based on visual self-calibration. The robot includes a robotic arm, a vision acquisition component, a station reference component, a composite end effector, an end effector status acquisition component, and a controller. The robotic arm drives the composite end effector between a first station and a second station. The vision acquisition component acquires images of the first station, the second station, and the composite end effector. The station reference component includes a first reference feature at the first station and a second reference feature at the second station. The composite end effector includes grippers, a suction cup, a lifting plate, and a guide plate. The grippers hold the material, the suction cup helps stabilize the material, the lifting plate supports the bottom of the material, and the guide plate limits material deviation near the loading inlet. The end effector status acquisition component acquires clamping force, suction negative pressure, gripper stroke, and tool posture. The controller is connected to the vision acquisition component, the robotic arm, the composite end effector, and the end effector status acquisition component, and performs image recognition, deviation estimation, candidate point scoring, trajectory planning, end effector compensation, and loading verification.

[0016] Optionally, the vision acquisition components include a fixed camera, an end-effector depth camera, a supplementary light, a dustproof transparent window, and an air knife cleaning component. The fixed camera is positioned facing the first and second workstations, while the end-effector depth camera moves with the composite end-effector and acquires local depth information when approaching the gripping point or loading inlet. The controller includes an image quality detection module, a reference benchmark recognition module, a material feature recognition module, a coordinate deviation estimation module, a candidate point scoring module, a trajectory planning module, an end-effector compensation module, a loading verification module, and an anomaly handling module. When the robot is positioned in a closed or dustproof workstation, the controller drives the air knife cleaning component to clean the dustproof transparent window based on the image quality detection results, and re-acquires images after cleaning.

[0017] The technical solution of this invention establishes an online self-calibration relationship during the material handling process from the first station to the second station, utilizing station reference benchmarks, material natural characteristics, loading inlet boundaries, and end-tool status. This calibrates whether the handling action is executed based on quality constraints, determines the current handling point using candidate point scoring, completes the handling, transfer, loading, and exiting processes using a continuous trajectory, and updates the compensation amount for the next cycle based on the post-loading verification results. This solution forms a unified control closed loop for material identification, coordinate calibration, handling point selection, end-tool status judgment, and loading verification.

[0018] Through the above technical solutions, the robotic arm can correct its gripping coordinates and trajectory based on the current image and end-effector status, even when there are conveyor belt deviations, equipment vibrations, changes in hopper or pallet replacements, changes in loading inlet position, and inconsistent material postures. This reduces the impact of static calibration failures or single-shot visual positioning errors on the gripping action. Simultaneously, candidate point scoring and end-effector status compensation ensure that the gripping position, clamping method, adsorption state, lifting state, and loading posture are matched, reducing the probability of empty gripping, off-center clamping, slippage, collision with the loading inlet, and incomplete loading. The visual, weight, material level, or end-effector residue verification results after loading are used to update deviation compensation and historical evaluation, allowing subsequent gripping cycles to utilize previous execution results for correction, thereby improving the stability and feasibility of the continuous material gripping process. Attached Figure Description

[0019] To more clearly illustrate the technical solution of this application, the accompanying drawings are briefly described below.

[0020] Figure 1 The flowchart of the material gripping and loading method based on visual self-calibration provided by the present invention is shown.

[0021] Figure 2 The structural diagram of the material vision self-calibration-based gripping robot system provided by the present invention is shown.

[0022] Figure 3 This is a diagram showing the layout of material handling application scenarios provided by the present invention.

[0023] Figure 4 The online visual self-calibration relationship diagram provided by this invention.

[0024] Figure 5 This invention provides a diagram for generating candidate loading points and planning continuous trajectories.

[0025] Figure 6 The composite end tool and end state feedback structure provided by the present invention are shown in the figure.

[0026] Figure 7 This is a diagram showing the relationship between post-loading verification and deviation update provided by the present invention.

[0027] Explanation of reference numerals in the attached figures: 100 is the gripping robot system; 110 is the robotic arm; 120 is the vision acquisition component; 121 is the fixed camera; 122 is the end-effector depth camera; 123 is the supplementary light; 124 is the dustproof transparent window; 125 is the air knife blowing component; 130 is the workstation reference component; 131 is the first reference feature; 132 is the second reference feature; 140 is the composite end-effector tool; 141 is the gripper; 142 is the suction cup; 143 is the lifting plate; 144 is the guide guard plate; 145 is the end-effector mounting base; 150 is the end-effector status acquisition component; 151 is the gripping force sensor; 152 is the negative pressure sensor; 1 53 is the gripper stroke sensor, 154 is the tool posture detection unit, 160 is the controller, 161 is the image quality detection module, 162 is the reference datum recognition module, 163 is the material feature recognition module, 164 is the coordinate deviation estimation module, 165 is the candidate point scoring module, 166 is the trajectory planning module, 167 is the end compensation module, 168 is the loading verification module, 169 is the anomaly handling module, 200 is the first station, 210 is the second station, 220 is the material to be gripped and loaded, 230 is the loading inlet, 240 is the conveyor line, 250 is the feeding hopper, and 260 is the weighing or material level feedback unit. Detailed Implementation

[0028] The technical solution of the present invention will be further described below with reference to the accompanying drawings. The described embodiments are used to explain the present invention, but are not intended to limit the present invention. The technical features of each embodiment can be combined with each other where there is no conflict.

[0029] Example 1

[0030] The following combination Figures 1 to 7 The material gripping method based on visual self-calibration of the present invention will be further described. This embodiment takes the gripping of bagged powder material from a conveyor line to a feeding hopper as an example. The conveyor line 240 constitutes the first station 200, the feeding hopper 250 constitutes the second station 210, the bagged powder material constitutes the material to be gripped 220, and the upper opening of the feeding hopper 250 constitutes the loading inlet 230. A first reference feature 131 is provided on the end side plate of the conveyor line 240. The first reference feature 131 can be a combination of two circular marks and a straight edge. A second reference feature 132 is provided near the outer edge of the loading inlet of the feeding hopper 250. The second reference feature 132 can be a ring boundary, a straight edge of the hopper opening, or a clean-resistant coded mark. The robotic arm 110 is positioned between the conveyor line 240 and the feeding hopper 250. The composite end tool 140 is installed at the end of the robotic arm 110. The fixed camera 121 is positioned facing the end of the conveyor line and the feeding hopper. The end depth camera 122 moves with the composite end tool 140 and acquires local depth images when it is close to the material or the loading inlet.

[0031] Reference Figure 1The method first executes step S1. After receiving the conveyor line arrival signal, the controller 160 controls the fixed camera 121 to acquire global images of the first station 200 and the second station 210, and reads the clamping force, suction negative pressure, gripper stroke, and initial tool posture output by the end-effector acquisition component 150. If the robot arm has just completed the loading action in the previous cycle, the controller 160 also reads the compensation amount of the previous cycle stored in the loading verification module 168. The images acquired by the fixed camera 121 are used to identify the end reference of the conveyor line, the outer contour of the bag, and the position of the feeding hopper. The end-effector depth camera 122 is used to supplement the information on the top height of the bag, the undulation of the bag surface, and the depth of the hopper before the robot arm 110 approaches the gripping area.

[0032] The image quality detection module 161 performs quality detection on the acquired image. The detection includes image sharpness, brightness uniformity, visibility of the first reference feature 131, visibility of the second reference feature 132, depth data integrity, and occlusion / contamination ratio. If the image has localized reflections but the first reference feature 131, the second reference feature 132, and the main boundary of the bag are still identifiable, the controller 160 marks the image as verifiable and reduces the subsequent approach speed of the robotic arm. If the dustproof transparent window 124 or dust on the bag surface makes the reference reference invisible, the controller 160 drives the air knife blowing component 125 to blow away the dustproof transparent window 124 and controls the supplementary light 123 to adjust the brightness before re-acquiring the image. If the re-acquisition still does not meet the quality requirements, the anomaly handling module 169 suspends the automatic gripping action.

[0033] In step S2, the reference identification module 162 identifies a first reference feature 131 and a second reference feature 132 from the qualified image. The first reference feature 131 is used to determine the actual end position, conveying direction, and lateral offset of the conveyor line 240. The second reference feature 132 is used to determine the loading center, loading plane, and accessible area of ​​the feeding hopper 250. The material feature identification module 163 identifies the outer contour, bag corner position, sealing direction, flat area of ​​the bag surface, top height, bulging area, and wrinkled area of ​​the material to be loaded 220. For bagged powder materials, areas with abrupt changes in bag height, severely wrinkled areas, and areas near the weak point of the bag opening are marked as low-priority gripping areas; areas with relatively flat bag surfaces, sufficient boundary margin, and close to the projection of the bag's center of gravity are marked as usable gripping areas.

[0034] In step S3, the coordinate deviation estimation module 164 estimates the deviation increment of the current gripping cycle based on the first reference feature 131, the second reference feature 132, and the material's natural characteristics. Let the camera coordinate system be C, the robot's base coordinate system be R, the first station coordinate system be F, the second station coordinate system be L, and the end-effector coordinate system be G. During initial debugging, the system saves T_RC0, T_RF0, T_RL0, and T_RG0, where T_RC0 represents the initial transformation relationship from the camera coordinate system C to the robot's base coordinate system R, T_RF0 represents the initial transformation relationship from the first station coordinate system F to the robot's base coordinate system R, T_RL0 represents the initial transformation relationship from the second station coordinate system L to the robot's base coordinate system R, and T_RG0 represents the initial transformation relationship from the end-effector coordinate system G to the robot's base coordinate system R. During the current loading cycle, the system calculates the deviation increment ΔT based on the difference between the current position and the theoretical position of the reference benchmark, and uses ΔT to correct the position of the material to be loaded 220 and the loading inlet 230 in the robot's base coordinate system R.

[0035] In this embodiment, the deviation increment ΔT can be obtained through a weighted fitting of the reference point set. Let the reference point identified in the current image be p_Ci, and its corresponding position in the theoretical workstation model be p_Ri. Then, the coordinate deviation estimation module 164 solves for ΔT that minimizes Σw_i||p_Ri-ΔT p_Ci||², where w_i is the confidence weight of the i-th reference point or boundary feature. If the deviation increment ΔT_{k-1} has been obtained in the previous cycle, a smoothing constraint λ||ΔT-ΔT_{k-1}||² can be added to the current solution process to avoid sudden changes in compensation due to misidentification in a single frame. The controller 160 simultaneously calculates the calibration quality Q, Q=αc_m+βc_r+γc_d+δc_l-ηc_o, where c_m is the confidence level of the material feature, c_r is the confidence level of the reference point identification, c_d is the depth data integrity, c_l is the confidence level of the loading entrance boundary, and c_o is the occlusion / contamination ratio. When the calibration quality reaches the normal gripping threshold, the system allows the robot to move at normal speed; when the calibration quality is within the verification range, the system enters low-speed gripping or secondary shooting; when the calibration quality is below the verification range, the system pauses automatic operation.

[0036] In step S4, the candidate point scoring module 165 generates a set of gripping candidate points G={g_1, g_2, ..., g_m} based on the outline of the bagged material, top height, bag surface flatness, and historical successful records. Each gripping candidate point g_i includes the gripping center, gripping posture, gripper opening, suction cup contact position, lifting plate entry direction, and loading posture. For bagged powder materials, the system prioritizes setting the suction cup contact area in the upper middle part of the bag where the bag surface is flat, and sets the gripper action line in positions with sufficient boundary margin on both sides of the bag, so that the lifting plate 143 can support the bag from below or below the side. If a candidate point is too close to a corner of the bag, located in a severely wrinkled area, or requires a large angle of rotation when the bag is loaded into the hopper, the score of that candidate point is reduced.

[0037] The score S(g_i) of the candidate grabbing point is calculated as S(g_i) = w_1V_i + w_2A_i + w_3B_i + w_4H_i + w_5R_i - w_6C_i - w_7D_i - w_8P_i. V_i represents the visibility of the candidate point, A_i represents the reachability of the robotic arm and end effector to the candidate point, B_i represents the boundary margin between the candidate point and the edge of the bag, H_i represents the high stability of the area, R_i represents the historical success rate for the same material or the same workstation, C_i represents the collision risk during approach and lifting, D_i represents the risk of bag deformation or slippage, and P_i represents the penalty for mismatch between the grabbing posture and the feeding hopper's inlet posture. Controller 160 selects the candidate point whose score satisfies the safety constraints and is ranked optimally as the current grabbing point.

[0038] In step S5, the trajectory planning module 166 plans a continuous trajectory based on the current gripping point, loading inlet boundary, and end-effector status. The continuous trajectory includes an approach segment, a gripping segment, a lifting segment, a transfer segment, a loading inlet alignment segment, a lowering loading segment, a release segment, and an exit segment. The approach segment causes the composite end-effector 140 to approach the current gripping point along the normal or approximately normal direction of the bag surface, preventing the gripper 141 from rubbing against the side of the bag during the approach. The gripping segment causes the suction cup 142 to first contact the bag surface and establish negative pressure, then the gripper 141 closes to a stroke adapted to the bag thickness, and subsequently the lifting plate 143 enters the support area below the bag. The lifting segment determines a safe lifting height based on the bag top height, the conveyor line side plate height, and surrounding obstacles. The transfer segment avoids interference areas with the conveyor line end, the outer wall of the feeding hopper, and the robotic arm itself. The inlet alignment section corrects the posture of the composite end tool 140 based on the hopper opening center and posture determined by the second reference feature 132, so that the bag enters the feeding hopper 250 at an angle adapted to the inlet's accessible area. The descent loading section controls the bag to descend to the set depth, the release section completes the placement in the sequence of suction cup depressurization, gripper opening, and lifting plate retraction, and the exit section leaves the second station in a direction that avoids the edge of the hopper opening.

[0039] In step S6, the robotic arm 110 performs a gripping action according to a continuous trajectory. During execution, the end effector compensation module 167 continuously reads the clamping force F_c, suction negative pressure P_v, gripper stroke L_j, tool posture angle θ_g, and slippage state s_g. After the suction cup 142 contacts the bag surface, if the suction negative pressure P_v does not reach the lower limit, the controller 160 can adjust the suction cup contact position or reduce the suction action speed. After the gripper 141 closes, if the clamping force F_c is lower than the clamping lower limit, or the gripper stroke L_j is inconsistent with the visually estimated bag thickness, the system determines that there is a risk of empty gripping or biased gripping. After the robotic arm 110 lifts the bag, if the tool posture angle θ_g changes abnormally or the slippage state s_g meets the slippage judgment condition, the system reduces the transfer speed and can return the bag to the first station for re-gripping. The above processing prevents the robotic arm from continuing to perform high-speed transfer and loading actions when the gripping state is unreliable.

[0040] In step S7, the loading verification module 168 acquires an image of the loading inlet of the feeding hopper 250 after the bag is released, and reads the weight or level information output by the weighing or level feedback unit 260. The verification includes whether the bag has entered the target area, whether the edge of the bag is caught on the hopper opening, whether there is end-effector residue on the composite end tool 140, whether the target area occupancy status is consistent with expectations, and whether the weighing or level change matches the current loading action. If the image shows a deviation of the bag from the theoretical loading position, the system calculates the error e = q_a - q_t between the actual loading position q_a and the theoretical loading position q_t, and decomposes this error into visual positioning error, end-effector error, and material slippage error based on the error direction and end-effector status record. The controller 160 updates the coordinate deviation compensation, end-effector compensation, and historical evaluation of the loading candidate points accordingly.

[0041] If the verification result shows that the bag has not fully entered the feeding hopper 250, the controller 160 can generate a secondary adjustment action based on the edge position of the bag, causing the guide plate 144 to gently push the edge of the bag or causing the lifting plate 143 to retract at a low speed and drive the bag into the target area. If the verification result shows that there is still material remaining at the end or the suction cup has not been fully released, the controller 160 performs suction cup depressurization confirmation and end-of-line shaking release actions. If the weighing or material level feedback is inconsistent with the visual verification result, the system marks the cycle as an abnormal sample that needs to be verified, and reduces the gripping speed of the next cycle or requires re-acquiring the loading inlet image. Through the above verification and update, this embodiment enables subsequent gripping cycles to use the previous loading error to correct the gripping point, release height, and loading inlet alignment posture.

[0042] In this embodiment, the equipment reference datum, the natural characteristics of the material, the loading inlet boundary, and the end-effector status all participate in the gripping control. Even if the conveyor line 240 shifts laterally relative to its theoretical position, or the feeding hopper 250 changes position due to vibration, cleaning, maintenance, or replacement of the hopper, the controller 160 can still estimate the deviation increment online using the first reference feature 131, the second reference feature 132, and the natural characteristics of the bagged material, and correct the gripping trajectory of the robotic arm 110. Compared with the method of performing pick-and-place actions solely based on static calibration coordinates, this embodiment can reduce the probability of bag misalignment, empty gripping, slippage after clamping, bag rubbing against the hopper opening, and incomplete loading, and enables the continuous gripping process to have self-calibration and update capabilities based on the verification results.

[0043] Example 2

[0044] This embodiment, based on Embodiment 1, further explains the specific implementation method of online visual self-calibration. This embodiment corresponds to... Figure 4 The online vision self-calibration relationship shown is mainly used to solve the problem that the robot can still correct the gripping coordinates based on the current vision results after relative offsets occur between the camera, robot arm, first station, second station and end tool during the production process.

[0045] Reference Figure 4 During the initial debugging phase, the system establishes fundamental transformation relationships between the camera coordinate system C, the robot's base coordinate system R, the first station coordinate system F, the second station coordinate system L, and the end-effector coordinate system G. These fundamental transformation relationships include T_RC0, T_RF0, T_RL0, and T_RG0. T_RC0 represents the initial transformation from the camera coordinate system C to the robot's base coordinate system R; T_RF0 represents the initial transformation from the first station coordinate system F to the robot's base coordinate system R; T_RL0 represents the initial transformation from the second station coordinate system L to the robot's base coordinate system R; and T_RG0 represents the initial transformation from the end-effector coordinate system G to the robot's base coordinate system R. These fundamental transformation relationships can be obtained through robot calibration, station reference measurement, and end-effector calibration, and serve as the initial state for subsequent online calibration.

[0046] At the start of the k-th gripping cycle, the fixed camera 121 acquires global images of the first station 200 and the second station 210, while the end-effector depth camera 122 acquires local depth images based on the current position of the robotic arm 110. The reference datum recognition module 162 identifies the first reference feature 131 and the second reference feature 132. The first reference feature 131 represents the current spatial state of the first station 200, and the second reference feature 132 represents the current spatial state of the second station 210. The material feature recognition module 163 simultaneously identifies the visible outline, top height, edge direction, and stable gripping area of ​​the material to be gripped 220, enabling the system to associate the fixed station reference with the current state of the material.

[0047] In one specific implementation, the first reference feature 131 includes two circular marks and a straight reference edge disposed on the side plate at the end of the conveyor line. The two circular marks are used to determine the planar position of the end of the conveyor line, and the straight reference edge is used to determine the conveying direction. The second reference feature 132 includes the arcuate boundary of the loading inlet of the feeding hopper 250 and a cleaning-resistant mark disposed on the outside of the loading inlet. The arcuate boundary of the loading inlet is used to fit the center of the loading inlet and the loading inlet plane, and the cleaning-resistant mark is used to provide an auxiliary reference when the loading inlet boundary is partially obscured by material. The above reference features maintain a fixed positional relationship with the corresponding equipment during the production process, so their current position in the image can reflect the offset of the equipment relative to the initial state.

[0048] The coordinate deviation estimation module 164 transforms the reference points, reference lines, or reference boundaries identified in the current image to the camera coordinate system C. For reference features with available depth data, the system directly obtains their three-dimensional coordinates; for reference features obtained only from two-dimensional images, the system combines the initial calibration plane, the workstation model, or the geometric constraints of the loading entrance to recover their spatial positions. Let the measured position of the i-th reference feature in the current cycle be p_Ci, and its corresponding theoretical position in the initial workstation model be p_Ri. The system solves for the deviation increment ΔT_k in the current cycle, ensuring that the current measured position, after deviation compensation, is consistent with the theoretical position. The deviation increment ΔT_k can be obtained according to the following objective function:

[0049] ΔT_k = argmin Σ w_i ||p_Ri - ΔT p_Ci||² + λ ||ΔT - ΔT_{k-1}||².

[0050] Where w_i is the confidence weight of the i-th reference feature, λ is the smoothing coefficient, and ΔT_{k-1} is the deviation increment obtained in the previous grasping cycle. When the reference feature is clear, the depth data is complete, and its geometric relationship with adjacent features is consistent, a higher weight is assigned to w_i; when the reference feature has local occlusion, reflection, or missing depth, a lower weight is assigned to w_i. Through the above processing, the system can utilize multiple reference features to jointly estimate the deviation and reduce the impact of misidentification of a single feature on the calibration results.

[0051] After the deviation increment is calculated in the current cycle, the system updates the coordinate transformation relationships. The current transformation relationship from the camera coordinate system to the robot base coordinate system can be expressed as T_RC,k = ΔT_RC,k · T_RC0, the current transformation relationship from the first station coordinate system to the robot base coordinate system can be expressed as T_RF,k = ΔT_RF,k · T_RF0, the current transformation relationship from the second station coordinate system to the robot base coordinate system can be expressed as T_RL,k = ΔT_RL,k · T_RL0, and the current transformation relationship from the end-effector coordinate system to the robot base coordinate system can be expressed as T_RG,k = ΔT_RG,k · T_RG0. For scenarios where only the overall offset of the workstation needs to be corrected, ΔT_RC,k, ΔT_RF,k, ΔT_RL,k, and ΔT_RG,k can be derived from the same deviation estimation process. For scenarios where the camera bracket, conveyor line, hopper, and end effector may each have an offset, the system calculates the corresponding deviation increment for each and combines them according to the coordinate chain during trajectory planning.

[0052] When material features are used in calibration, they do not replace the equipment reference datum, but are used to verify whether the deviation estimation result is consistent with the actual position of the material to be picked up. The system determines the material coordinate system M based on the material outline, top height, and edge direction, and transforms the material coordinate system M into the robot's base coordinate system R. If the area where the material is located, calculated from the equipment reference datum, is consistent with the actual area of ​​the material obtained by visual recognition, the current deviation increment is confirmed to be valid; if the deviation exceeds the allowable range, the system re-checks the reference datum recognition result, depth data integrity, and material segmentation result. In cases where the material bag is partially obscured or has localized reflection, the system prioritizes using the equipment reference datum to determine the station offset, and then uses the visible boundary of the material to correct the picking point, rather than directly using the incomplete material outline to correct all coordinate relationships.

[0053] When the loading entrance boundary is involved in calibration, the system extracts the edge line, arc, or corner point of the loading entrance 230 from the image of the second station 210, and calculates the loading entrance center, loading entrance plane, and accessible area. If the loading entrance boundary and the second reference feature 132 are both visible, the system uses both to estimate the deviation of the second station; if the loading entrance boundary is partially obscured by a material bag, flap, or flexible connector, the system determines the loading entrance posture based on the second reference feature 132 and the unobstructed loading entrance boundary. After the loading entrance center and loading entrance plane are transformed to the robot's base coordinate system R, they are used to correct the key trajectory nodes of the descent loading and release sections.

[0054] To determine whether the online calibration results can be used for automated loading, controller 160 calculates the calibration quality Q. The calibration quality Q can be determined according to the following relationship:

[0055] Q = αc_m + βc_r + γc_d + δc_l - ηc_o.

[0056] Wherein, c_m is the confidence level of material features, c_r is the confidence level of reference benchmark recognition, c_d is the completeness of depth data, c_l is the confidence level of loading entrance boundary, c_o is the proportion of occlusion and contamination, and α, β, γ, δ, and η are weighting coefficients. The confidence level of material features c_m is determined based on the integrity rate of the material outline, the stability of the top height, and the continuity of the edges; the confidence level of reference benchmark recognition c_r is determined based on the recognition integrity rate and geometric consistency of the first benchmark feature 131 and the second benchmark feature 132; the completeness of depth data c_d is determined based on the proportion of effective depth points and the depth noise level; the confidence level of loading entrance boundary c_l is determined based on the visible length of the loading entrance edge, the fitting residual, and the stability of the center position; and the proportion of occlusion and contamination c_o is determined based on the proportion of dust, window contamination, reflective saturation area, and occlusion area to the key area of ​​the image.

[0057] When the calibration quality Q reaches the normal gripping threshold, the controller 160 sends the updated coordinate transformation relationship to the candidate point scoring module 165 and the trajectory planning module 166, and the robotic arm performs the gripping action at a normal speed. When the calibration quality Q is within the verification range, the controller 160 controls the robotic arm 110 to reduce its approach speed and controls the fixed camera 121 or the end-effector depth camera 122 to perform secondary acquisition; if the secondary acquisition result is consistent with the first acquisition result, the system allows low-speed gripping. When the calibration quality Q is below the verification range, the anomaly handling module 169 suspends automatic gripping and triggers supplementary lighting adjustment, air knife purging, re-identification of reference benchmark, or prompts manual processing based on the image quality detection result.

[0058] This embodiment also includes event-triggered calibration. Event-triggered calibration is performed when changing batches, changing bag specifications, changing pallets, changing material buckets, replacing end-of-line tools, opening and closing isolation doors, restarting the conveyor after an emergency stop, resetting the feeding hopper after cleaning, or when continuous gripping failures or continuously increasing loading deviations occur. Compared to ordinary periodic calibration, event-triggered calibration involves collecting more frames, has a lower robot arm movement speed, and requires that the geometric relationships between the first reference feature 131, the second reference feature 132, and the material's natural characteristics simultaneously meet consistency conditions. If the event-triggered calibration fails, the system does not directly resume automatic gripping.

[0059] In this embodiment, the results of online visual self-calibration are used not only to correct the coordinates of the gripping point, but also to correct the alignment posture of the loading inlet, end-effector compensation, and the compensation amount for the next cycle. For example, if the verification information shows that the bag shifts in the same direction every time it enters the feeding hopper 250, the system allocates this shift amount to the second station deviation compensation and release posture compensation; if the clamping force, suction negative pressure, and tool posture are all normal, but the loading position is still offset, the system prioritizes updating the compensation related to the second station or loading inlet; if loading deviation and tool posture abnormalities occur simultaneously, the system prioritizes updating the end-effector compensation and lowers the historical evaluation of such gripping candidate points. Through the above methods, a continuous feedback relationship is formed between visual self-calibration, end-effector status, and loading verification, so that the coordinate compensation comes not only from the image recognition before gripping, but also from the execution results after gripping.

[0060] Example 3

[0061] This embodiment, based on Embodiments 1 and 2, further explains the specific implementation methods of candidate point generation, scoring, and continuous trajectory planning. This embodiment corresponds to... Figure 5 The relationship between candidate grabbing points and continuous trajectory planning shown is mainly used to illustrate how the system can determine the appropriate grabbing point for the current cycle from multiple possible grabbing positions when the material shape is not completely consistent, the loading inlet position is deviated, and the end tool state is different, and form a continuous trajectory that can complete grabbing and loading.

[0062] Reference Figure 5 The material feature recognition module 163 first performs feature segmentation on the material 220 to be picked up. For bagged powder materials, the system extracts the outer contour of the bag, bag corners, sealing direction, and visible boundary from the global image of the fixed camera 121, and obtains the top height of the bag, local bulges, sagging areas, and surface flatness from the data output by the end depth camera 122 or depth acquisition unit. The system divides the interior of the outer contour of the bag into multiple candidate regions and eliminates regions located in areas with severe wrinkles, obvious reflections, edge occlusion, weak areas at the bag opening, or excessive height changes. In the remaining regions, the positions with high bag surface flatness, a safe margin from the edge of the bag, and close to the projection of the material's center of gravity are identified as candidate picking point source areas.

[0063] For drummed materials, the system extracts the outer contour of the drum, the arc of the drum opening, the direction of the drum sidewalls, the handle area, and the tilt angle of the drum. Candidate gripping points are preferentially generated in the upper-middle part of the drum where the gripper's action line is close to the drum's center of gravity projection, avoiding weak edges of the lid, the handle's active area, and reflective areas of the label. For boxed or palletized materials, the system extracts corner points, straight edges, holes, grids, and support surfaces. Candidate gripping points are preferentially generated in locations with clear edges, sufficient gripping margin, and consistent with the loading direction. For lumpy materials formed after sieving or cooling, the system determines a stable gripping area based on the height map, outer contour, and surrounding gaps, and generates candidate actions for first straightening and then gripping when necessary.

[0064] Each gripping candidate point g_i includes not only a spatial location, but also the corresponding gripping posture, approach direction, gripper opening, adsorption position, lifting position, and loading posture. For bagged powder materials, the gripping posture is determined based on the local normal of the bag surface, the sealing direction, and the long and short sides of the bag; the approach direction is usually consistent with the normal of the bag surface or the approximate normal of a locally flat surface; the gripper opening is determined based on the local thickness of the bag and the packaging flexibility; the adsorption position is located in a relatively flat area of ​​the bag surface; the lifting position is located below or slightly below the center of gravity projection of the bag; and the loading posture is determined based on the shape of the inlet of the feeding hopper 250 and the size of the bag. Therefore, a gripping candidate point represents a local scheme of a complete gripping action, rather than a single image coordinate point.

[0065] The candidate point scoring module 165 calculates a comprehensive score S(g_i) for each grabbing candidate point g_i. The comprehensive score can be determined according to S(g_i)=w_1V_i+w_2A_i+w_3B_i+w_4H_i+w_5R_i-w_6C_i-w_7D_i-w_8P_i. V_i is visibility, calculated based on the visible proportion of the candidate point's area in the image, boundary continuity, and the proportion of effective depth points; A_i is end-effector accessibility, calculated based on the joint range of motion of the robotic arm 110, the posture limitations of the composite end-effector tool 140, and whether the approach direction is executable; B_i is boundary margin, calculated based on the distance from the candidate point to the outer contour of the material, bag edge, barrel edge, or pallet boundary; H_i is height stability, calculated based on the height variance, local slope, and support stability within the candidate area; R_i is historical success rate, calculated based on successful records under the same material type, the same workstation, and similar gripping postures; C_i is collision risk, calculated based on the distance between the approach path and lifting path of the composite end-effector tool 140 and the conveyor line, hopper, isolation cover, or surrounding materials; D_i is material deformation or slippage risk, calculated based on soft bag sagging, barrel tilting, box surface strength, or the degree of clumping looseness; P_i is inlet posture mismatch penalty, calculated based on the degree of matching between the material posture after gripping and the inlet accessible area of ​​the second workstation 210.

[0066] In the scoring calculation, the weights w_1 to w_8 can be adjusted according to the material category and workstation requirements. For soft-bag materials, the system increases the weight corresponding to Di, reducing the ranking of candidate points in areas prone to sagging, slippage, or weak packaging. For drummed materials, the system increases the weights corresponding to A_i, B_i, and P_i, making the clamping line, boundary allowance, and filling inlet posture matching relationship have a greater impact on the results. For materials after sieving or cooling, the system increases the weights corresponding to H_i and C_i, making high stability and collision risk the main screening factors. For weighing and batching stations, the system increases the weights corresponding to R_i and P_i, making historical success rate and landing point control participate in the gripping point selection. The above weight adjustments can be given by preset process parameters or can be smoothly updated based on historical records.

[0067] After calculating the comprehensive score, the system does not directly select the candidate point with the highest score, but first performs a safety constraint check. Safety constraints include whether the candidate point is within the material gripping area, whether the composite end effector 140 can reach the candidate point in a set posture, whether the approach path interferes with the equipment boundary, whether the shape of the material after gripping can pass through the loading inlet 230 of the second station 210, whether the gripper 141 and lifting plate 143 exceed the accessible area of ​​the loading inlet during loading, and whether the robotic arm 110 meets speed, load, and obstacle avoidance requirements throughout the entire movement. Only candidate points that simultaneously meet both safety constraints and scoring ranking requirements can be determined as the current gripping point.

[0068] Once the current gripping point is determined, the trajectory planning module 166 generates a continuous trajectory. The continuous trajectory includes a proximity segment, a gripping segment, a lifting segment, a transfer segment, a loading inlet alignment segment, a descent loading segment, a release segment, and an exit segment. The proximity segment begins with the current pose of the robotic arm 110, causing the composite end effector 140 to enter the gripping area along the proximity direction corresponding to the candidate point. This segment operates at a low speed and corrects the final contact position based on the local depth information from the end effector depth camera 122. The gripping segment executes a combination of actions: suction cup 142 contact, negative pressure establishment, gripper 141 closure, and lifting plate 143 entry. For bagged materials, the gripping segment preferably first uses the suction cup 142 to adhere to the bag surface to stabilize the bag, then the gripper 141 clamps both sides of the bag, and finally the lifting plate 143 supports the bottom or lower side of the bag.

[0069] The lifting section is used to detach the material from the first station 200. The system determines the lifting height based on the top height of the material, the boundary of the first station equipment, and surrounding obstacles. If the end-effector status acquisition component 150 detects a decrease in clamping force F_c, fluctuation in adsorption negative pressure P_v, or abnormal change in tool attitude angle θ_g during the initial lifting phase, the trajectory planning module 166 pauses the transition to the transfer section, and the end-effector compensation module 167 determines whether re-clamping, adding lifting support, or releasing the gripper is necessary. By performing early status confirmation during the lifting section, the system can detect clamping abnormalities as soon as the material leaves the first station.

[0070] The transfer section is used to move materials from the first station 200 to above the second station 210. This section avoids the end of the conveyor line, the outer wall of the hopper, the boundary of the isolation chamber, surrounding containers, and areas where the robotic arm itself may interfere. The trajectory planning module 166 corrects the endpoint of the transfer section based on the deviation increment ΔT obtained in step S3, so that the robotic arm 110 does not simply reach the second station according to the initial calibration position, but reaches the alignment area according to the actual position of the loading inlet in the current cycle. For narrow stations, the transfer section can be set with an intermediate transition posture, so that the long side of the material first avoids the edge of the equipment before turning into the loading inlet alignment posture.

[0071] The loading inlet alignment section is generated based on the loading inlet boundary of the second station 210. The system uses the second reference feature 132 and the visible edge of the loading inlet 230 to determine the loading inlet center, loading inlet plane, accessible area, and prohibited collision area. For rectangular hopper openings, the system can determine the center and entry direction based on the four sides of the hopper opening; for circular barrel openings, the system can fit the barrel opening center based on the arc boundary; for loading inlets with flexible connections or sealing rings, the system uses the inner boundary of the flexible connection or the inner edge of the sealing ring as the accessible area boundary. The trajectory planning module 166 enables the composite end tool 140 to reach a posture adapted to the accessible area above the loading inlet, and the guide guard 144 is located on the side prone to collision or material snagging to limit the offset of the bag edge or material edge.

[0072] The descent loading section controls the material to enter the second station 210. The descent depth is determined based on the material size, loading inlet depth, target material level, and the shape of the composite end tool 140. For bagged materials, the system can use an inclined descent method, allowing the front end of the bag to enter the feeding hopper 250 first, and then the rear end to slowly exit with the lifting plate 143 and fall into the target area. For drummed or boxed materials, the system can use a vertical descent method, reducing speed as it approaches the target position. For bulk lumpy materials, the system can control the drop height and release posture to prevent the material from impacting the edge of the target container. In the descent loading section, the end depth camera 122 can continue to acquire local depth images of the loading inlet to confirm the distance between the material and the edge of the loading inlet.

[0073] The release phase is executed according to the material type and end-effector status. For bagged materials, the system can first reduce the negative pressure of the suction cup 142, then gradually open the gripper 141, while simultaneously causing the lifting plate 143 to exit in the opposite direction to the bottom of the bag, allowing the bag to enter the target area under gravity. For drummed or boxed materials, the system can first confirm that the bottom of the material is close to the bearing surface, then open the gripper 141 and hold it briefly to prevent the material from tipping over. After the release phase is completed, the exit phase leaves the second station 210 in a direction that avoids the edge of the loading inlet, and returns the composite end-effector 140 to an observable or standby position. If the loading verification module 168 determines that the loading position deviates from the target, the system can insert a secondary adjustment action before the exit phase.

[0074] In this embodiment, the scoring results of candidate gripping points and the continuous trajectory planning results are mutually constrained. Even if a candidate point has high gripping stability, it will not be selected as the current gripping point if its corresponding loading posture cannot pass through the loading inlet 230, or if there is an unavoidable collision risk in the continuous trajectory. Conversely, if a candidate point does not have the highest local gripping score, but it enables the material to enter the second station 210 in a more suitable posture, and the continuous trajectory is shorter and the collision risk is lower, the system can select this candidate point as the current gripping point. In this way, the selection of candidate points is not limited to the moment of gripping, but serves the complete gripping task from the first station 200 to the second station 210.

[0075] This embodiment also utilizes the historical success rate R_i to update the gripping strategy. After each gripping operation, the loading verification module 168 writes the gripping point position, gripping posture, end-effector status, loading deviation, and success status into the historical record. For candidate points that are successfully completed and have small loading deviations, the historical success rate of their similar areas increases in subsequent cycles; for candidate points that experience slippage, material snagging, empty gripping, or large loading deviations, the historical success rate of their similar areas decreases. The historical success rate is updated using a sliding window or exponential smoothing method to avoid significant fluctuations in the gripping strategy caused by a single anomaly. Thus, the system can gradually correct the candidate point ranking based on the on-site execution results while maintaining rule constraints.

[0076] Example 4

[0077] This embodiment, based on the above embodiments, further illustrates the gripping robot based on material vision self-calibration. This embodiment combines... Figure 2 , Figure 3 and Figure 6 The structure shown describes the composition, module functions, specific implementation methods, and collaborative working process of the gripping robot system 100. The gripping robot system 100 includes a robotic arm 110, a vision acquisition component 120, a workstation reference component 130, a composite end effector 140, an end effector status acquisition component 150, and a controller 160. The robotic arm 110 drives the composite end effector 140 to move between the first workstation 200 and the second workstation 210. The vision acquisition component 120 acquires images of the first workstation 200, the second workstation 210, and the composite end effector 140. The workstation reference component 130 provides a recognizable workstation reference for online self-calibration. The composite end effector 140 performs gripping, adsorption, lifting, and guiding of the material to be gripped 220. The end effector status acquisition component 150 acquires the status of the composite end effector 140 during the gripping process. The controller 160 is used to perform image quality inspection, reference benchmark recognition, material feature recognition, coordinate deviation estimation, candidate point scoring, trajectory planning, end compensation, loading verification, and anomaly handling.

[0078] Reference Figure 2The robotic arm 110 can be a six-axis robotic arm, a four-axis robotic arm, a gantry robot, or a collaborative robotic arm. The selection of the robotic arm 110 is determined based on the distance between the first workstation 200 and the second workstation 210, the weight of the material, cycle time requirements, space limitations of the loading inlet 230, and cleanliness or dustproof requirements. The robotic arm 110 receives trajectory commands sent by the controller 160 and moves along a continuous trajectory of approaching, gripping, lifting, transferring, aligning, lowering for loading, releasing, and exiting. The robotic arm 110 can be configured with joint status feedback, collision detection, and speed limiting functions. When the controller 160 determines that the workstation is not ready, the loading inlet is not visible, the end effector is abnormal, or the safety interlock is triggered, the robotic arm 110 stops entering the gripping action or stops continuing to execute the current gripping action.

[0079] The vision acquisition component 120 includes a fixed camera 121, an end-effector depth camera 122, a supplementary light 123, a dustproof transparent window 124, and an air knife blowing component 125. The fixed camera 121 is fixedly positioned above or to the side of the first station 200 and the second station 210, used to acquire global images of the station and identify the first reference feature 131, the second reference feature 132, the outer contour of the material to be grasped 220, and the boundary of the loading inlet 230. The end-effector depth camera 122 is installed near the composite end tool 140 and moves with the robotic arm 110, used to acquire local depth information when near the grasping point or loading inlet 230, including the material top height, bag surface undulation, barrel opening height, loading inlet depth, and the distance between the end tool and the material. The supplementary light 123 is used to improve image brightness and boundary recognition stability. The dustproof transparent window 124 is used to separate the camera or optical window from dust, moisture, or cleanroom isolation spaces. The air knife blowing component 125 is used to blow away dust from the transparent window 124 when the window is contaminated or covered by dust.

[0080] The station reference component 130 includes a first reference feature 131 and a second reference feature 132. The first reference feature 131 is disposed on the first station 200 and maintains a fixed positional relationship with the first station 200; the second reference feature 132 is disposed on the second station 210 and maintains a fixed positional relationship with the loading inlet 230. In the conveyor line grabbing scenario, the first reference feature 131 can be disposed on the side plate, end baffle, or support frame of the conveyor line 240; in the screening equipment or cooler discharge scenario, the first reference feature 131 can be disposed on the outer frame of the screen, the edge of the discharge end, the corner of the tray, or the fixed structure of the cooler outlet; in the hopper or barrel loading scenario, the second reference feature 132 can be disposed on the outer edge of the hopper opening, the positioning ring of the barrel opening, the sealing ring fixing seat, or the bearing platform. The first reference feature 131 and the second reference feature 132 can be circular marks, geometric holes, straight edges, corner points, bosses, clean-resistant coding marks, or natural boundaries of the equipment. The controller 160 determines the deviations of the first station 200 and the second station 210 from the initial calibration state by identifying these reference features.

[0081] Reference Figure 6 The composite end effector 140 includes grippers 141, suction cups 142, a support plate 143, a guide plate 144, and an end mount 145. The end mount 145 is flange-connected to the robotic arm 110 and provides a mounting reference for the grippers 141, suction cups 142, support plate 143, and guide plate 144. The grippers 141 are used for lateral gripping of bagged, drummed, boxed, or pallet-loaded materials. The suction cups 142 are used to establish auxiliary suction on bag surfaces, thin sheets, or relatively flat surfaces to stabilize the material before or during the closing of the grippers 141. The support plate 143 is used to support the material from below or from the side when gripping soft bags or sagging materials, reducing sagging or slippage during transfer. The guide guard 144 is used to form a guide boundary when the composite end tool 140 approaches the loading inlet 230, so that the material to be loaded 220 enters the loading inlet 230 in the permitted direction and reduces the friction between the material edge and the loading inlet edge.

[0082] The specific structure of the composite end tool 140 can be adjusted according to the material type. In the case of bagged powder materials, the grippers 141 can be relatively flexible grippers with a cleanable anti-slip pad on the gripping surface. The suction cup 142 is located between or above the grippers 141, the lifting plate 143 is located below the grippers 141 and can enter the bottom support position of the bag after gripping, and the guide plate 144 is located at the front end of the lifting plate 143 or the grippers 141. For barrelled materials, the grippers 141 can have an arc-shaped gripping surface to adapt to the outer wall of the barrel, the suction cup 142 can serve as an auxiliary stabilizer, and the lifting plate 143 can be retracted or not participate in the main support. For boxed or palletized materials, the grippers 141 can have a parallel gripping surface, the lifting plate 143 is used to support the bottom edge, and the guide plate 144 is used to limit lateral displacement during placement.

[0083] The end-effector state acquisition component 150 includes a clamping force sensor 151, a negative pressure sensor 152, a gripper stroke sensor 153, and a tool posture detection unit 154. The clamping force sensor 151 is located near the drive mechanism or clamping surface of the gripper 141 and is used to detect the clamping force F_c applied by the gripper 141 to the material 220 to be gripped. The negative pressure sensor 152 is connected to the negative pressure channel of the suction cup 142 and is used to detect the suction negative pressure P_v of the suction cup 142. The gripper stroke sensor 153 is used to detect the closing stroke L_j of the gripper 141 to determine whether the actual closing position of the gripper is consistent with the visually estimated material size. The tool posture detection unit 154 can be composed of a posture sensor, end-effector encoding information, or visual recognition results, and is used to acquire the tool posture angle θ_g and the sliding state s_g. The output of the end-effector state acquisition component 150 is sent to the controller 160, which uses it to determine the gripping reliability and generate end-effector compensation actions.

[0084] The controller 160 includes an image quality detection module 161, a reference datum recognition module 162, a material feature recognition module 163, a coordinate deviation estimation module 164, a candidate point scoring module 165, a trajectory planning module 166, an end-effector compensation module 167, a loading verification module 168, and an anomaly handling module 169. The image quality detection module 161 evaluates the sharpness, brightness, reference visibility, depth integrity, and contamination / occlusion of the images output by the fixed camera 121 and the end-effector depth camera 122. The reference datum recognition module 162 identifies the first reference feature 131 and the second reference feature 132 and outputs the current measurement position of the workstation reference. The material feature recognition module 163 identifies the contour, edge direction, top height, flat area, deformed area, and occluded area of ​​the material 220 to be loaded. The coordinate deviation estimation module 164 estimates the deviation increment based on the workstation reference component 130, the material's natural features, and the loading entrance boundary. The candidate point scoring module 165 generates loading candidate points and calculates a comprehensive score. The trajectory planning module 166 is used to generate a continuous trajectory. The end-effector compensation module 167 is used to correct the gripping action based on clamping force, negative pressure, stroke, tool posture, and sliding state. The loading verification module 168 is used to determine whether the material is in place after loading and to generate error feedback. The anomaly handling module 169 is used to handle situations such as insufficient image quality, insufficient calibration quality, clamping abnormalities, invisible loading entrance, unready target station, and safety interlock triggering.

[0085] During system operation, the image quality detection module 161 first determines whether the current image can be used for gripping control. If the image is qualified, the reference datum recognition module 162 and the material feature recognition module 163 output the station reference position, material natural features, and loading entrance boundary, respectively. The coordinate deviation estimation module 164 calculates the deviation increment and calibration quality based on the above information. When the calibration quality meets the requirements, the candidate point scoring module 165 generates the current gripping point, the trajectory planning module 166 generates a continuous trajectory, and the robotic arm 110 drives the composite end effector 140 to perform the gripping action. During execution, the end effector status acquisition component 150 continuously outputs clamping force, suction negative pressure, gripper stroke, and tool posture to the end effector compensation module 167. After loading, the loading verification module 168 collects the image, material level, or weight information of the second station 210 and feeds back the verification results to the coordinate deviation estimation module 164 and the candidate point scoring module 165.

[0086] When the image quality detection module 161 detects contamination of the dustproof transparent window 124, dust obstruction, or image overexposure, the controller 160 drives the air knife blowing component 125 and the supplementary light 123 to process the issue and re-acquire the image. When the end-effector compensation module 167 determines that the clamping force is insufficient, the negative pressure is insufficient, the gripper stroke is abnormal, or the material slips, the controller 160 can control the robotic arm 110 to reduce its speed, re-grip, release the gripper, or switch to a backup gripping point. When the loading verification module 168 determines that the material is not fully loaded or is stuck at the edge of the loading entrance, the controller 160 can generate a secondary adjustment action and write the failure reason into the history record. Through the coordination between the above modules, the gripping robot system 100 can combine station visual recognition, coordinate self-calibration, end-effector status judgment, and post-loading verification into the same control process.

[0087] The gripping robot described in this embodiment can be installed as an independent workstation between the conveying and feeding equipment, or it can be used in combination with screening equipment, cooling equipment, weighing and batching systems, or closed isolation equipment. When used in a weighing and batching system, the second station 210 can be equipped with a weighing or material level feedback unit 260, and the controller 160 adjusts the next gripping amount, release height, or landing point based on the weighing or material level feedback results. When used in a closed isolation equipment, a fixed camera 121, an end-effector depth camera 122, and a composite end effector 140 can be arranged inside the isolation chamber, and a dustproof transparent window 124, a supplementary light 123, and an air knife blowing component 125 are used to maintain identifiable image quality. None of the above combinations change the basic structure of the gripping robot's gripping control through visual self-calibration.

[0088] Example 5

[0089] This embodiment further illustrates the specific application process of the present invention by combining an automatic feeding and weighing batching station for bagged powder materials. (Refer to...) Figure 3 and Figure 7In this embodiment, bagged powder material is conveyed to the first station 200 via conveyor line 240. A gripping robot system 100 grips the material 220 to be gripped via conveyor line 240 and loads it into the feeding hopper 250. A weighing or level feedback unit 260 is installed below the feeding hopper 250. The bagged powder material can be a soft-bag package with a net weight of 10 kg to 25 kg, and the bag's external dimensions can be 450 mm × 300 mm × 80 mm to 750 mm × 450 mm × 180 mm. The effective width of the conveyor line 240 can be 600 mm to 900 mm, and lateral offset, longitudinal offset, and angular deflection are allowed after the bag is in position. The loading inlet 230 of the feeding hopper 250 can be a rectangular opening or a rounded rectangular opening, and the effective size of the loading inlet can be set from 500 mm × 400 mm to 900 mm × 650 mm depending on the bag size. The dimensions and weights described above are only for illustrating one feasible application scenario and do not constitute a limitation on the scope of protection of this invention.

[0090] In this embodiment, the first reference feature 131 is disposed on the fixed side plates on both sides of the end of the conveyor line 240, including two clean-resistant circular marks and a straight edge extending along the conveying direction. The center distance between the two clean-resistant circular marks can be set to 300 mm to 600 mm to determine the translation and rotation state of the end of the conveyor line in the image; the straight edge is used to determine the conveying direction and lateral offset. The second reference feature 132 is disposed near the outer edge of the loading inlet 230 of the feeding hopper 250, including a straight edge of the outer edge of the hopper opening, a rounded corner boundary, and a clean-resistant coded mark. The outer edge of the hopper opening is used to fit the center of the loading inlet and the loading inlet plane, and the coded mark is used to provide an auxiliary positioning reference when the hopper opening is partially obstructed by a bag, dust, or a flap.

[0091] In terms of hardware configuration, the robotic arm 110 can be a six-axis robotic arm with a rated load of 30 kg to 50 kg, and its working radius covers the end of the conveyor line 240, the area above the feeding hopper 250, the vision verification position, and the standby position. The fixed camera 121 can be an industrial area array camera with a resolution of 3 million to 12 million pixels, and its installation height can be 1.5 m to 2.4 m, with the lens field of view covering the end of the conveyor line and the top of the feeding hopper. The end-effector depth camera 122 is installed above or to the side front of the composite end tool 140, with a working distance of 300 mm to 1200 mm, and is used to obtain the bag surface height, hopper opening depth, and local distance between the end and the material. The supplementary light 123 can be a ring or strip light source, with the illumination direction avoiding the main reflective direction of the bag surface. The air knife blowing component 125 can be set with a blowing pressure of 0.2 MPa to 0.5 MPa and a blowing time of 1 s to 5 s, and is used to treat dust adhering on the dustproof transparent window 124.

[0092] The composite end tool 140 adopts a combination structure of grippers 141, suction cups 142, lifting plates 143, and guide plates 144. The opening stroke of the grippers 141 can be set to 250 mm to 500 mm, and the gripping surface is provided with a removable anti-slip pad; the suction cups 142 can be set between the grippers 141, and the working range of adsorption negative pressure P_v can be set to -30 kPa to -70 kPa; the lifting plates 143 can enter the bottom or lower side of the bag after gripping, and their extension length can be set to 120 mm to 300 mm; the guide plates 144 are set on the side of the composite end tool 140 near the loading inlet, and are used to limit the lateral displacement of the bag edge when the bag enters the feeding hopper 250. The clamping force sensor 151 is used to collect the clamping force F_c. The lower limit of the clamping force can be set from 80 N to 250 N according to the weight of the bag. The gripper stroke sensor 153 is used to determine whether the actual closing stroke is consistent with the visually estimated bag thickness. The tool posture detection unit 154 is used to detect the posture change of the bag relative to the end tool during the transfer process.

[0093] During system operation, the conveyor line 240 delivers bagged materials to the first station 200 and then stops. The controller 160 receives the conveyor line arrival signal and triggers the fixed camera 121 to acquire global images of the first station 200 and the second station 210. The image quality detection module 161 calculates image clarity, brightness uniformity, visibility of the first reference feature 131, visibility of the second reference feature 132, depth data integrity, and occlusion / contamination ratio. For example, when the confidence level of the first reference feature recognition is higher than 0.85, the confidence level of the loading entrance boundary is higher than 0.80, the proportion of effective depth points in the key area is higher than 0.75, and the occlusion / contamination ratio is lower than 0.20, the system enters the normal loading process; when the above indicators are in a low but still verifiable range, the system reduces the speed of the robotic arm and performs a second acquisition; when the reference reference is not visible or the occlusion / contamination ratio is too high, the controller 160 first drives the air knife blowing component 125 to blow away the dustproof transparent window 124, and then re-acquires the image.

[0094] After the image is deemed acceptable, the reference recognition module 162 identifies the reference at the end of the conveyor line and the reference at the feeding hopper opening, while the material feature recognition module 163 identifies the bag outline, bag corners, sealing direction, flat areas of the bag surface, bulging areas, wrinkled areas, and the current posture of the bag. The coordinate deviation estimation module 164 calculates the deviations of the end of the conveyor line and the feeding hopper opening relative to the initial calibration state based on the first reference feature 131 and the second reference feature 132. For example, when it is identified that the end of the conveyor line is laterally offset by 18 mm relative to the initial position, the center of the bag is offset by 42 mm relative to the center of the conveyor line, the bag is deflected by 6° relative to the conveying direction, and the center of the feeding hopper opening is offset by 12 mm relative to the initial position, the controller 160 converts the above deviations into deviation increments in the robot's base coordinate system and simultaneously corrects the gripping point, alignment point, and descent loading point.

[0095] The candidate point scoring module 165 generates multiple gripping candidate points in flat areas of the bag surface, locations with clamping allowances on both sides of the bag, and historically successful gripping areas. For each candidate point, the system calculates visibility, end-point reachability, boundary allowance, height stability, historical success rate, collision risk, material deformation or slippage risk, and penalty for mismatch in the loading inlet posture. For example, candidate point g_1 is located in the area slightly to the left of the bag center, with good bag surface flatness but close to the sealing edge; candidate point g_2 is located in the central area of ​​the bag, with a complete suction cup contact surface and smooth entry direction of the lifting plate; candidate point g_3 is near the corner of the bag, with small clamping allowance but good matching of the loading inlet posture. After comprehensive scoring, if g_2 has the highest score and meets the constraints of gripper opening, suction cup contact area, lifting plate entry space, and loading inlet passability, then g_2 is determined as the current gripping point.

[0096] The trajectory planning module 166 then generates a continuous trajectory. In the approach phase, the robotic arm 110 brings the composite end effector 140 close to the bag surface at low speed, and the end depth camera 122 acquires local depth images and corrects the final contact position. In the gripping phase, the suction cup 142 first contacts the bag surface and establishes negative pressure, for example, reaching -45 kPa. Then, the gripper 141 closes to a stroke matching the visually estimated bag thickness, and the lifting plate 143 extends into the lower side of the bag. In the lifting phase, the robotic arm 110 first lifts the bag 80 mm to 150 mm for gripping confirmation. If the gripping force F_c, suction negative pressure P_v, and tool attitude angle θ_g are all within the allowable range, the lifting continues to a safe height. If the gripping force is below the set lower limit, the suction negative pressure fluctuation exceeds the allowable range, or the gripper stroke differs significantly from the visually estimated thickness, the controller 160 pauses the transfer and performs a re-gripping or release re-gripping.

[0097] In the transfer section, the robotic arm 110 moves the bag above the feeding hopper 250 according to the corrected second station coordinates. In the loading inlet alignment section, the end depth camera 122 again captures a local depth image of the hopper opening, and the controller 160 corrects the bag's entry posture according to the second reference feature 132 and the loading inlet boundary. For rectangular hopper openings, the system can maintain a preset angle between the long side of the bag and the long side of the hopper opening; for cases where the bag width is close to the hopper opening width, the system can allow the front end of the bag to enter the loading inlet 230 first, and then slowly withdraw the lifting plate 143, with the guide plate 144 preventing the side of the bag from rubbing against the edge of the hopper opening. In the release section, the system first reduces the negative pressure of the suction cup 142, then gradually opens the gripper 141, and controls the lifting plate 143 to withdraw in the opposite direction to the bag's falling direction, causing the bag to fall into the feeding hopper 250.

[0098] After loading is completed, the loading verification module 168 collects the image of the hopper opening and reads the data from the weighing or material level feedback unit 260. For example, if the theoretical weight of a single bag is 20 kg, the increase in weighing feedback is between 19.6 kg and 20.4 kg, and the image shows that the bag has completely entered the target area, then this loading record is considered a qualified execution; if the increase in weighing is significantly lower than the theoretical value and the end image shows residue near the suction cup or lifting plate, the system performs an end release confirmation; if the image shows that the edge of the bag is caught in the hopper opening, the system generates a secondary adjustment action, causing the guide plate 144 or the lifting plate 143 to push the edge of the bag into the hopper at a low speed; if the actual loading position q_a deviates from the theoretical loading position q_t by more than a set value, for example, more than 30 mm, the system writes the error e=q_a-q_t into the next cycle placement point compensation and reduces the historical evaluation of the candidate point that caused the deviation.

[0099] This embodiment can also set historical update rules for continuous production processes. For the same material specification, the same conveyor line location range, and the same loading inlet status, the system saves the most recent 50 to 300 grab records. Each record includes material contour characteristics, grab candidate point number, grab posture, clamping force, adsorption negative pressure, gripper stroke, loading deviation, weighing change, and verification result. For multiple consecutive loading deviations with the same direction, the system allocates the deviation to loading inlet coordinate compensation or release posture compensation; for multiple consecutive slippages in the same bag surface area, the system increases the weight of material deformation or slippage risk D_i; for a candidate area with consistently small loading deviations, the system increases the historical success rate R_i of that area. Through this historical update process, the grab strategy can adapt to differences in on-site equipment status and material packaging without changing the basic control flow.

[0100] The processing effect of this embodiment can be illustrated by the following table. The values ​​in the table are example data to illustrate the processing logic of this invention. Actual values ​​can be adjusted according to equipment size, bag weight, production cycle time, and environmental requirements.

[0101] Lateral offset of conveyor line The conveyor line end is offset by 18 mm, and the bag center is offset by 42 mm. Following the initial bag-retrieving point, the grippers may deviate from the bag's stress area. The first baseline feature and the bag contour are used together to refine the grab points, and the candidate points are updated according to the current bag position. Bag posture deflection The bag body is deflected 6° relative to the conveying direction. The gripping direction remains fixed, resulting in uneven force distribution on the gripping surface. Adjust the gripper posture and the lifting plate entry direction according to the sealing direction and the long side of the bag. Partial wrinkles on the bag surface There is a bulge in the middle of the bag and folds along the edges. The center point may be selected, but the suction cup negative pressure is unstable. Reduce the score for wrinkled and highly abrupt areas, and prioritize areas that are flat and have sufficient margin at the boundaries. Hopper opening offset The hopper opening center is offset by 12 mm, and there is local dust obstruction at the boundary. The placement point is still executed in the initial position, which may cause contact with the hopper opening. The second reference feature and the visible loading port boundary together correct the alignment point and descent path. Abnormal clamping state Negative pressure not reaching -35 kPa or abnormal gripper stroke Delayed anomaly detection may lead to further transfer. Pause lifting or transfer, re-adsorb, re-clamp, or release for re-grabbing. Loading deviation repeat The feed was loaded into the hopper several times in succession, with the feed position biased to one side. The next cycle will still be executed at the original placement point. The loading error is allocated to placement point compensation, release attitude compensation, and candidate point history evaluation.

[0102] In this embodiment, the gripping robot system 100 can complete material gripping even when there are fluctuations in the conveyor line position, bag posture, hopper opening boundary, end-effector gripping state, and loading result. It follows a sequence of image quality detection, online coordinate self-calibration, candidate point scoring, continuous trajectory planning, end-effector state compensation, and post-loading verification and updating. This process not only determines whether the material can be gripped but also whether it can enter the feeding hopper 250 in a suitable posture. It also uses the deviation after loading for correction in the next cycle, making it suitable for continuous feeding and weighing / dispensing linkage scenarios.

[0103] Example 6

[0104] This embodiment further illustrates the application of the present invention in continuous discharge and container loading scenarios, specifically at the station where material is gripped from the cooler's discharge end to the transfer container. In this embodiment, the first station 200 is the discharge end of a drum-type cooler or similar cooling conveying equipment, and the second station 210 is a transfer container, material box, or subsequent conveying pallet. The cooler discharge end typically suffers from problems such as inconsistent material arrival positions, varying discharge heights depending on the stacking state, materials sticking together, localized dust obstruction, and equipment vibration at the discharge end. The transfer container may also deviate from its theoretical placement position during replacement, pushing, or positioning. Therefore, this embodiment utilizes the first reference feature 131, the second reference feature 132, the material height map, and the container opening boundary to achieve visual self-calibration gripping.

[0105] In this embodiment, the first reference feature 131 can be set on the fixed side plate, discharge port bracket, or end stop of the conveyor belt at the discharge end of the cooler, including a straight reference edge and two temperature-resistant calibration blocks. The straight reference edge is used to determine the direction of the discharge end, and the temperature-resistant calibration blocks are used to determine the planar position and local rotational deviation of the discharge end. The second reference feature 132 is set on the outer edge of the turnover container or the container positioning platform, including the container corner, container edge line, or positioning pin hole. The fixed camera 121 is arranged facing the discharge end of the cooler and the turnover container, and the end depth camera 122 is used to acquire local depth information when close to the material pile and the container opening. If the material still has a certain temperature after cooling, the gripper 141 and lifting plate 143 of the composite end tool 140 can adopt a temperature-resistant contact surface, and the gripping surface can be provided with a replaceable anti-slip pad to reduce the influence of dust or temperature difference on the material surface on the gripping state.

[0106] When the system starts working, the cooler conveys the material to the discharge end, and the controller 160 determines the material's arrival status based on signals from upstream equipment. After the fixed camera 121 acquires images of the discharge end, the image quality detection module 161 determines whether the first reference feature 131 is clear, whether the material at the discharge end is located in the grabbing area, and whether the opening of the turnover container is visible. In cases where there is dust or localized steam at the discharge end, if the key reference reference is still identifiable and the depth data integrity meets the verification requirements, the system can reduce the approach speed of the robotic arm 110; if the reference reference is not visible, the system triggers purging, waits for dust to settle, or pauses grabbing. The reference reference identification module 162 identifies the straight reference edge and the temperature-resistant calibration block at the cooler's discharge end, and the coordinate deviation estimation module 164 calculates the deviation of the first station 200 relative to the initial calibration state accordingly.

[0107] The material feature recognition module 163 generates a height map of the discharge end based on the fixed camera image and the depth data output by the end depth camera 122. The height map is used to distinguish between individual material units, material contact areas, void areas, and high-pile areas that may collapse. For relatively regular block or box-shaped materials, the system extracts their outer contour, corner points, and main direction; for irregularly shaped or closely packed clumps, the system extracts the local highest point, surrounding slope, supporting area, and clamping boundary. If the height of a certain area varies too much, the surrounding void is insufficient, or it is close to the discharge end baffle, the system marks that area as a low-priority candidate area; if the height of a certain area is stable, the boundary is clear, and the gripper 141 has sufficient entry space, the system uses it as a source of gripping candidate points.

[0108] In this embodiment, the system increases the weights of height stability H_i, collision risk C_i, and boundary margin B_i when scoring candidate points. For example, when the height difference between the highest area of ​​the material pile and its surroundings exceeds a preset value and the slope is large, this area may slip when the grippers approach, and the candidate point's D_i and C_i increase accordingly. When the minimum gap between the outer boundary of the material and adjacent materials is greater than the sum of the single-sided thickness of the gripper 141 and the safety margin, the candidate point's A_i and B_i increase accordingly. For scenarios requiring multiple small pieces of material to be gripped and loaded into a transfer container at once, the system can also select to grip a group of materials or adopt a candidate action of first separating and then gripping, based on the remaining space in the container and the target loading distribution. The execution of these candidate actions is still determined through unified scoring and safety constraint checks.

[0109] The trajectory planning module 166 generates a continuous trajectory based on the current gripping point and the opening of the turnover container. The approach section guides the composite end tool 140 into the discharge end in a direction that does not interfere with adjacent materials. The gripping section selects a clamping, lifting, or clamping and lifting method based on the material's shape. For single blocky materials, the grippers 141 clamp the material from both sides and detect the clamping force F_c during the initial lifting phase. For flatter materials or materials with unstable bottom support, the lifting plate 143 enters the support position from below after clamping. For materials that are partially close together, the robotic arm 110 can first perform a low-speed straightening action to create a gripper space between the material to be gripped and adjacent materials. The lifting section prioritizes a small-height trial lifting method, for example, lifting 30 mm to 80 mm first and detecting the clamping force, tool posture, and material displacement relative to the end of the material. Once stability is confirmed, the material enters the transfer section.

[0110] When the container is loaded, the second reference feature 132 and the container opening boundary are used to correct the current coordinates of the second station 210. If the container is laterally offset by 20 mm relative to the theoretical position, or if the container opening is slightly deflected due to incomplete loading, the coordinate deviation estimation module 164 converts the deviation to the robot's base coordinate system R, and the trajectory planning module 166 corrects the loading inlet alignment section and the descent loading section accordingly. For containers that require layered or zoned loading, the loading verification module 168 updates the occupancy status inside the container after each placement. If a certain area has reached the target height or is close to the container edge, the target landing point for the next grabbing and loading will be shifted to the unoccupied area to avoid local stacking or offset.

[0111] After loading, the loading verification module 168 judges the loading result through container opening image, depth and height map, and weighing or material level feedback. If the material landing position deviates little from the theoretical landing point and the container occupancy status meets the distribution requirements, the system records the loading as successful and improves the historical success rate of the corresponding candidate point and landing point combination. If the material accumulates near the edge of the container after release, the system records the deviation as landing point compensation; if the material does not completely detach from the composite end tool 140 or flips during release, the system writes the end tool status and release posture as abnormal reasons into the history record. For situations where the height of the turnover container gradually increases during continuous production, the system dynamically adjusts the release height according to the material level or height map inside the container to avoid the material falling too far or colliding with existing materials.

[0112] This embodiment demonstrates the applicability of the invention in scenarios involving irregular material delivery and container loading. Compared to methods requiring upstream equipment to neatly arrange materials before grabbing, this embodiment utilizes a discharge end reference baseline, material height map, candidate point scoring, and container verification results to determine the grabbable area, enabling the robotic arm to adapt to changes in material position and accumulation caused by continuous discharge from the cooler. Compared to methods that simply place containers in fixed positions, this embodiment corrects the landing point and next cycle compensation amount based on the actual opening position of the turnover container, the container occupancy status, and post-loading verification results, reducing the risks of material stacking misalignment, contact with container edges, and repeated grabbing failures.

[0113] In other embodiments, the visual acquisition component 120 can be a combination of a fixed camera 121 and an end-effector depth camera 122, or it can be a single RGB-D camera covering the first station 200 and the second station 210. When there are highly reflective packaging or metal containers on site, a polarization component can be set in front of the supplementary lighting 123, or a combination of a structured light camera, a binocular camera, a line laser profilometer, and an area array camera can be used for acquisition. For stations with high dust levels, the dustproof transparent window 124 can be equipped with an air knife blowing component 125, a micro-positive pressure protection structure, or a replaceable transparent sheet. As long as the visual acquisition component can obtain the equipment reference benchmark, the natural characteristics of the material, and the loading entrance boundary, it can be used for the online self-calibration gripping control of this invention.

[0114] In other embodiments, the first reference feature 131 and the second reference feature 132 can be a coding mark, a dot array, a geometric hole, a straight edge, a corner point, a boss, a tray corner, a screen frame, a hopper boundary, or a hopper positioning ring. For cleanroom equipment where it is not suitable to affix or install explicit markings, the stable geometric boundary of the equipment itself can be used as a reference. For workstations that require frequent cleaning or sterilization, the reference feature can be a clean-resistant, corrosion-resistant, and non-detachable uneven structure or a machined boundary.

[0115] In other embodiments, the composite end effector 140 can be adjusted according to the material type. For soft-bag materials, a combination of grippers 141, suction cups 142, and lifting plates 143 can be used; for barrelled materials, a combination of curved grippers and anti-slip pads can be used; for boxed materials, a combination of parallel grippers and bottom lifting components can be used; for pallet-borne materials, a combination of grippers, insert plates, or positioning pins can be used; and for lumpy or bulk materials, a combination of grippers, shovels, or quantitative grabbers can be used. The end effector status acquisition component 150 can also be selected based on the tool type, using clamping force, negative pressure, stroke, posture, vibration, weight, or contact switch signals. As long as the end effector status can be used to determine clamping reliability and feed back to the controller 160, the end effector compensation function of this invention can be realized.

[0116] In other implementations, post-loading verification can be performed individually or in combination by visual verification, weighing verification, material level verification, end-of-pipe residue detection, container occupancy detection, or downstream equipment arrival signals. For scenarios where the loading inlet is obstructed by material and direct visual observation is inconvenient, weighing or material level signals can be prioritized; for scenarios requiring control of placement posture, loading inlet images and target area occupancy status can be prioritized; for scenarios where material may adhere to the end-of-pipe tool, end-of-pipe residue detection can be added. Verification results can be used to update coordinate deviation compensation, end-of-pipe tool compensation, and candidate point historical evaluation.

[0117] In other embodiments, the present invention can be used for bagged powder entering the feeding hopper from a conveyor line, for materials from the outlet of a cooler entering a transfer container, for screened materials entering a weighing container, for drummed materials entering an isolation chamber carrier, for boxed materials entering a stacking pallet, and for scenarios involving multiple robotic arms working together to handle materials. When multiple robotic arms work together, each robotic arm can share the reference benchmarks and historical evaluation data of the first and second workstations, but each maintains its own end-effector compensation parameters and motion safety zone.

[0118] The embodiments described above are used to illustrate the technical solutions of the present invention, and are not intended to limit the present invention. Those skilled in the art can adjust or replace the visual acquisition method, reference benchmark method, end-effector structure, scoring weight, verification signal source, and application station without departing from the concept of the present invention. Any material handling method and robot that performs online self-calibration based on equipment reference benchmarks, material natural characteristics, loading inlet boundaries, and end-effector status, and accordingly completes the determination of candidate loading points, continuous trajectory control, and post-loading verification and updating, should fall within the equivalent scope of the technical solutions of the present invention.

Claims

1. A material handling method based on visual self-calibration, characterized in that, include: S1, acquire images and status information of the first station, the second station, and the end effector of the robot, and perform quality inspection on the images; S2, when the quality inspection meets the loading conditions, identify the equipment reference benchmark on the first station, the material natural characteristics of the material to be loaded, and the loading entrance boundary of the second station. S3, based on the equipment reference datum, the natural characteristics of the material and the loading port boundary, estimate the deviation increment between the camera coordinate system, the robot arm base coordinate system, the first station coordinate system, the second station coordinate system and the end tool coordinate system online, and form the calibration quality; S4. Based on the deviation increment and calibration quality, generate material loading candidate points, score the loading candidate points and determine the current loading point; S5 plans a continuous trajectory for grabbing, transferring, aligning, loading, and exiting based on the current grabbing point, loading boundary, and end-effector status. S6, control the robotic arm to perform gripping actions according to the continuous trajectory, and collect the end-effector status during the execution; S7 collects verification information after loading, and updates the deviation increment, historical evaluation of loading candidate points, and action compensation amount for the next cycle based on the verification results.

2. The material handling method based on visual self-calibration according to claim 1, characterized in that, In step S2, the equipment reference datum includes a first reference feature fixed on the first station and a second reference feature fixed on the second station. The first reference feature is used to determine the positional deviation of the station where the material to be loaded is located relative to the theoretical station. The second reference feature is used to determine the loading inlet center, loading inlet posture, and the accessible area of ​​the loading inlet. The material natural features include the material outline, material top height, material edge direction, material surface flatness, and material occlusion area. The system associates the equipment reference datum with the material natural features in the same frame or adjacent frames to avoid relying solely on offline calibration results to generate loading coordinates.

3. The material handling method based on visual self-calibration according to claim 1, characterized in that, The quality inspection in step S1 includes evaluating image sharpness, brightness uniformity, reference visibility, depth data integrity, and occlusion / contamination ratio. When the image quality is lower than the first quality threshold but higher than the second quality threshold, the robot arm is controlled to enter a low-speed verification state and re-acquire the image. When the image quality is lower than the second quality threshold, the gripping action is paused, and supplementary lighting adjustment, visual window cleaning, or abnormal prompts are triggered. The re-acquired image re-enters step S2 to prevent mis-grabbing caused by dust, reflection, window contamination, or the target being out of the field of view.

4. The material handling method based on visual self-calibration according to claim 1, characterized in that, In step S3, the initial coordinate transformation relationship is saved, and the deviation increment ΔT is calculated within the current loading cycle to minimize the weighted distance between the current measured position of the reference benchmark and the corresponding theoretical position. The calibration quality Q is calculated based on the material characteristic confidence level c_m, the reference benchmark identification confidence level c_r, the depth data integrity level c_d, the loading entrance boundary confidence level c_l, and the occlusion and contamination ratio c_o, satisfying Q=αc_m+βc_r+γc_d+δc_l-ηc_o, where α, β, γ, δ, and η are weighting coefficients. Automatic loading is performed when the calibration quality reaches the normal loading threshold, low-speed loading or secondary shooting is performed when the calibration quality is in the verification range, and automatic loading is stopped when the calibration quality is below the verification range.

5. The material handling method based on visual self-calibration according to claim 1, characterized in that, In step S4, the candidate grabbing points are generated from the stable region within the material outline, the reinforced region at the material edge, the flat region at the top of the material, and the historically successful grabbing region. Each candidate grabbing point is associated with spatial location, grabbing posture, approach direction, clamping opening, adsorption position, lifting position, and loading posture. The score S(g_i) of the candidate grabbing point is calculated according to S(g_i)=w_1V_i+w_2A_i+w_3B_i+w_4H_i+w_5R_i-w_6C_i-w_7D_i-w_8P_i, where V_i is visibility, A_i is end-point reachability, B_i is boundary margin, H_i is high stability, R_i is historical success rate, C_i is collision risk, D_i is material deformation or slippage risk, and P_i is the penalty for mismatch in loading posture. The system selects the candidate grabbing point whose score meets the safety constraints and has the best ranking as the current grabbing point.

6. The material handling method based on visual self-calibration according to claim 1, characterized in that, In step S5, the continuous trajectory includes an approach segment, a gripping segment, a lifting segment, a transfer segment, a loading inlet alignment segment, a descent loading segment, a release segment, and an exit segment. The approach segment determines the approach angle based on the normal or edge direction of the current gripping point. The lifting segment determines the safe height based on the material height and surrounding obstacles. The loading inlet alignment segment corrects the robot's posture based on the loading inlet center, loading inlet posture, and accessible area of ​​the second station. The descent loading segment limits the descent depth based on the material size and end-effector shape. The exit segment leaves the second station along a direction that avoids the loading inlet edge. When a new deviation increment is obtained in step S3, the key trajectory nodes of the continuous trajectory are corrected synchronously with the deviation increment.

7. The material handling method based on visual self-calibration according to claim 1, characterized in that, In step S6, the end-effector state includes clamping force F_c, adsorption negative pressure P_v, gripper stroke L_j, tool posture angle θ_g, and slip state s_g. After the robot approaches the current gripping point, it first enters the contact state at a low speed, then closes the gripper and establishes adsorption negative pressure according to the material size. Subsequently, it judges whether the gripping is reliable by the clamping force, adsorption negative pressure, and gripper stroke. When the clamping force is lower than the lower limit of clamping, the adsorption negative pressure is lower than the lower limit of negative pressure, the gripper stroke is inconsistent with the visually estimated material size, or the tool posture angle changes abnormally, the system determines that the current gripping has the risk of empty gripping, biased gripping, or slippage, and performs release, re-grip, speed reduction, or switching to a backup gripping point.

8. The material handling method based on visual self-calibration according to claim 1, characterized in that, In step S7, the verification information includes the image of the loading entrance of the second station, the material loading position, the occupancy status of the target area, the material level information, the weight information, and the end-of-line residue status. When the verification result shows that the material deviates from the target loading position, the error e=q_a-q_t between the actual loading position q_a and the theoretical loading position q_t is decomposed into visual positioning error, end-of-line tool error, and material slippage error, and the historical evaluation of coordinate deviation compensation, end-of-line tool compensation, and grab candidate points is updated respectively. When the verification result shows that the material is not fully loaded, there is material hanging on the edge of the loading entrance, or there is end-of-line residue, a secondary adjustment action or a re-grabbing action is generated, and the corresponding failure reason is written into the history record.

9. A gripping robot based on material vision self-calibration, characterized in that, Includes robotic arm, vision acquisition component, workstation reference component, composite end effector, end effector status acquisition component and controller; The robotic arm is used to drive the composite end effector to move between the first and second workstations. The vision acquisition component is used to acquire images of the first workstation, the second workstation, and the composite end effector. The workstation reference component includes a first reference feature disposed at a first workstation and a second reference feature disposed at a second workstation; The composite end tool includes a gripper, a suction cup, a lifting plate, and a guide plate. The gripper is used to hold the material, the suction cup is used to help stabilize the material, the lifting plate is used to support the bottom of the material, and the guide plate is used to limit the material's deviation near the loading inlet. The end-effector acquisition component is used to acquire clamping force, suction negative pressure, gripper stroke, and tool posture. The controller is connected to the vision acquisition component, the robotic arm, the composite end effector, and the end effector status acquisition component, and is configured to estimate the coordinate deviation increment based on the reference benchmark of the image recognition device, the natural characteristics of the material, and the loading port boundary, generate candidate points for gripping and loading and continuous trajectories, and control the robotic arm to complete gripping, transferring, loading and verification updates.

10. The material vision-based self-calibration gripping robot according to claim 9, characterized in that, The vision acquisition component includes a fixed camera, an end-effector depth camera, a supplementary light, a dustproof transparent window, and an air knife cleaning component. The fixed camera is positioned facing the first and second workstations. The end-effector depth camera moves with the composite end-effector and is used to acquire local depth information when approaching the gripping point or loading inlet. The controller includes an image quality detection module, a reference benchmark recognition module, a material feature recognition module, a coordinate deviation estimation module, a candidate point scoring module, a trajectory planning module, an end-effector compensation module, a loading verification module, and an anomaly handling module. Each module works collaboratively in the order of image quality detection, reference benchmark and material feature recognition, deviation estimation, candidate point scoring, trajectory planning, end-effector state compensation, loading verification, and anomaly handling. When the robot arm is positioned in a closed or dustproof workstation, the controller drives the air knife cleaning component to clean the dustproof transparent window based on the image quality detection results and re-acquires images after cleaning.

Images