A visual-haptic fusion closed-loop safe grasping method and system for fragile porcelain plates

Abstract

The application discloses a kind of fragile porcelain plate-oriented visual haptics fusion closed-loop safety grabbing method and system, belong to robot intelligent grabbing, machine vision and haptics perception control technical field.The method obtains porcelain target and work area information by RGB-D image, and the porcelain target is segmented, and the pose of work area or reference object is estimated to establish work coordinate reference;According to the candidate grabbing pose generated by target mask and depth information, the mechanical arm is controlled to approach the porcelain target at low speed after safety screening and coordinate transformation;In the process of gripper closure, clamping confirmation, post-grabbing reinforcement, first-stage lifting and placement, stable contact, clamping state and slip risk are judged using haptics data.The application can reduce the risk of fragile porcelain plate slipping, colliding and being damaged by force while ensuring the stability of grabbing, and is suitable for cultural relic protection, fragile object transfer and robot flexible operation scenarios.

Description

Technical Field

[0001] This invention relates to the fields of robot intelligent grasping, machine vision, tactile perception and visual-tactile fusion control technology, specifically to a visual-tactile fusion closed-loop safe grasping method and system for fragile ceramic plates. Background Technology

[0002] In tasks such as underwater shipwreck archaeology, cultural relic preservation, and the transfer of fragile objects, porcelain plates are typically characterized by their thin structure, fragile edges, smooth surfaces, and sensitivity to stress. In particular, ancient porcelain that has been immersed in seawater or buried in sediment for a long time may have aged, become porous, or become brittle in its glaze, body, and edge structure. During extraction and transfer, it is extremely easy for improper stress to cause cracks, chipping, or even complete breakage.

[0003] Traditional manual extraction methods rely heavily on operator experience. While feasible in certain scenarios, their operational stability and repeatability are difficult to guarantee. In underwater, confined, murky, or complex stacked environments, manual operation also suffers from low efficiency, high risk, and difficulty in precisely controlling gripping forces. Therefore, using robots to replace manual labor for grasping and placing fragile ceramic discs has significant application value.

[0004] Existing robotic arm grasping methods typically include manual teaching, offline trajectory planning, and vision-based intelligent grasping. Among these, manual teaching and offline trajectory planning methods generally require pre-setting the robotic arm's motion path, gripper opening width, and clamping force parameters, making them suitable for structured scenarios where the target position and posture are relatively fixed. However, in grasping fragile targets like ceramic discs, the target's placement, posture, occlusion, and support status are often uncertain, making fixed grasping strategies difficult to adapt to changes in the actual environment. If the clamping force is too small, the ceramic disc is prone to slipping; if the clamping force is too large, it may damage the disc's edges or main structure.

[0005] With the development of machine vision and deep learning technologies, vision-based robotic grasping methods are increasingly being applied to tasks such as target detection, instance segmentation, pose estimation, and grasping posture generation. Vision methods can acquire information about the target's position, contour, depth, and spatial pose, providing a basis for the robotic arm to approach the target and plan the grasping posture. For example, target instance segmentation models can be used to extract the porcelain plate region, six-DOF pose estimation methods can be used to determine the spatial pose of the work area or reference object, and grasping posture prediction networks can be used to generate candidate grasping poses suitable for a two-finger gripper.

[0006] However, relying solely on vision for grasping still has limitations. First, the vision system primarily reflects the target's position, shape, and posture before grasping, making it difficult to directly perceive the actual contact state between the gripper and the ceramic disc. For fragile ceramic discs, it's difficult to accurately determine whether stable contact has been established, whether local slippage has occurred, or whether the gripping force exceeds safe limits based solely on visual information. Second, visual recognition and depth perception are easily affected by occlusion, reflections, lack of depth information, weak edges, and changes in ambient lighting, leading to discrepancies between the grasping pose and the actual executable posture. Third, the grasping posture output by the visual grasping network is usually only a candidate result before grasping. Without a feedback correction mechanism during the grasping process, the robotic arm may still slip, deviate, or be damaged during the contact, closing, lifting, and placement phases.

[0007] Tactile sensing technology can directly acquire physical interaction information between the gripper and the target. By setting multi-dimensional tactile sensors at the end of the gripper, information such as the number of contact points, the distribution of contact areas, normal force, tangential force, pressure changes, and slippage trends can be detected in real time. Compared with visual perception, tactile perception can provide more direct safety judgment after the gripper contacts the target, which helps to achieve adaptive adjustment of gripping force, stable contact determination, and abnormal state handling.

[0008] However, relying solely on tactile perception is insufficient to complete a full robotic grasping task. Tactile information is typically only obtained after the gripper makes contact with the target, failing to provide the target's global position, orientation, and optimal approach direction before grasping. Therefore, without visual localization and grasping posture prediction, the robotic arm struggles to accurately approach the target and select a suitable grasping point. While some existing vision-tactile fusion methods utilize both visual and tactile information simultaneously for grasping tasks, they often focus on a single stage of perception or control, failing to establish a closed-loop control system encompassing target recognition, work area localization, grasping posture generation, contact detection, gripping adjustment, grasping reinforcement, lifting verification, and safe placement.

[0009] Therefore, existing technologies suffer from at least the following problems: traditional fixed-position grasping methods struggle to adapt to the uncertain changes in the position, posture, and support state of ceramic disc-like targets; purely visual grasping methods lack real-time perception of contact force, slippage state, and gripping safety, easily leading to object slippage and damage; purely tactile grasping methods lack global positioning and posture planning capabilities before grasping, making it difficult to independently grasp targets in complex scenarios; and existing vision-and-tactile fusion grasping methods typically lack continuous safety control throughout the entire process of grasping, securing, lifting, and placing. Based on these problems, it is necessary to propose a safe grasping method and system that collaboratively integrates target segmentation, work area pose estimation, grasping posture prediction, robotic arm motion control, gripper control, and tactile feedback adjustment. Summary of the Invention

[0010] This invention addresses the challenges of positioning, grasping, controlling gripping force, and slippage / breakage during robotic grasping and placement of fragile porcelain plates. It proposes a vision-tactile fusion closed-loop safe grasping method and system for fragile porcelain plates, resolving issues such as unstable target positioning, inconsistent coordinate references in the work area, difficulty in directly executing grasping posture candidates, lack of contact status feedback in purely visual methods, lack of coordination between tactile feedback and the motion process, and lack of safety verification during the lifting and placement phases after grasping. This invention is applicable to scenarios such as underwater shipwreck archaeology, cultural relic preservation, transfer of fragile objects, and handling of precision components, and is particularly suitable for automated robotic grasping and placement of easily slippery and fragile targets such as porcelain plates, ceramic plates, glass plates, and thin-walled vessels. The visual perception module acquires spatial information about the work area and the ceramic plate target to be grasped. Combined with the grasping posture prediction module, an executable grasping posture is generated. The tactile sensor obtains the contact state, force distribution and slippage risk between the gripper and the ceramic plate in real time. Then, the gripper closing, grasping reinforcement, lifting and placement processes are controlled in a closed loop, so as to achieve stable, safe and low-damage grasping of fragile ceramic plate targets.

[0011] To address the aforementioned technical problems, this invention provides a visual-tactile fusion closed-loop safe gripping method for fragile ceramic plates. This method acquires spatial information about the ceramic plate target and the working area through visual perception, obtains suitable candidate gripping postures for two-finger gripper execution through gripping posture prediction, and introduces tactile feedback during the robotic arm's gripping process to make real-time judgments on contact state, gripping stability, and slippage risk, thereby achieving safe gripping, stable lifting, and reliable placement of the fragile ceramic plate target.

[0012] The method includes: acquiring color and depth images of the work scene through a vision acquisition module; performing instance segmentation on the ceramic plate target to obtain target mask, bounding box, category confidence, and target area depth information; performing six-degree-of-freedom pose estimation on the work area or work reference object, and establishing a coordinate reference for the work area in conjunction with the calibration relationship between the camera and the robot; generating a local point cloud of the target or grasping input data based on the ceramic plate target area and its depth information, and generating multiple candidate grasping poses; performing safety screening on the candidate grasping poses and converting them into execution poses in the robot's base coordinate system; controlling the robotic arm to move to the pre-grasping position and approaching the ceramic plate target at low speed; synchronously acquiring tactile data during the low-speed closing of the gripper, and determining whether stable contact has been formed based on the tactile data; performing gripping confirmation and post-grasping reinforcement after stable contact; verifying the risk of slippage during the first stage of low-speed lifting; and transporting the ceramic plate to the placement area and placing it safely at low speed after successful verification.

[0013] This invention also provides a visual-tactile fusion closed-loop safety grasping system for fragile porcelain plates. The system includes a visual acquisition module, a target segmentation module, a work area pose estimation module, a grasping posture prediction module, a coordinate transformation module, a robot execution module, a tactile perception module, and a safety control module. The visual acquisition module acquires color and depth images of the work scene; the target segmentation module identifies and extracts the target area of ​​the porcelain plate; the work area pose estimation module acquires the six-degree-of-freedom pose of a sandbox, pallet, calibration board, or work reference object and establishes a work coordinate reference; the grasping posture prediction module generates candidate grasping postures based on the target area and depth information; the coordinate transformation module converts the grasping posture in the camera coordinate system into the execution posture in the robot base coordinate system; the robot execution module includes a robotic arm and a two-finger gripper for performing approach, gripping, lifting, transporting, and placement actions; the tactile perception module acquires contact information between the gripper and the porcelain plate; and the safety control module performs closed-loop control of the grasping process based on visual results, tactile information, and gripper status.

[0014] Compared with existing technologies, this invention has at least the following beneficial effects: First, by identifying and segmenting the ceramic disc target through a vision module and establishing a coordinate reference for the work area using a six-degree-of-freedom pose estimation method, the accuracy of positioning the fragile ceramic disc target and performing pose transformations by the robotic arm can be improved. Second, instead of directly gripping based on the target's center point, the system generates multiple candidate gripping poses based on the target segmentation results, depth information, and gripping posture prediction results. It then comprehensively considers factors such as gripping score, maximum gripper opening, target mask range, depth consistency, gripping direction, and robotic arm reachability for selection, thereby improving the rationality of gripping posture selection. Third, by setting a tactile sensing module at the gripper end to detect the number of contact points, contact area connectivity, total contact force, and continuous stability of the contact state in real time, the shortcomings of simple vision-based gripping, which lacks contact feedback, can be compensated for. Fourth, through gripping confirmation, post-grip reinforcement, and a first-stage low-speed lifting verification, unstable gripping states can be detected before the ceramic disc completely leaves the support surface, reducing the risk of falling, collisions, and secondary damage. Fifth, visual information is mainly used for pre-grasping localization, posture generation, and path guidance, while tactile information is mainly used for contact confirmation, gripping stability judgment, and abnormal state detection during grasping. The two complement each other, enabling the system to have both global localization capabilities and local contact perception capabilities. Attached Figure Description

[0015] Figure 1 This is a block diagram of the overall system structure of the present invention;

[0016] Figure 2 This is a schematic diagram of the hardware scenario setup for the present invention;

[0017] Figure 3This is a schematic diagram of the target segmentation result of the ceramic plate according to the present invention;

[0018] Figure 4 This is a schematic diagram illustrating the operation area identification and segmentation of the present invention;

[0019] Figure 5 This is a schematic diagram illustrating the pose estimation and coordinate reference establishment of the working area in this invention;

[0020] Figure 6 This is a schematic diagram illustrating the generation and selection of candidate grasping postures in this invention;

[0021] Figure 7 This is a schematic diagram of the robotic arm's low-speed approach and gripping state according to the present invention;

[0022] Figure 8 This is a schematic diagram illustrating the contact state and slippage risk determination based on tactile data according to the present invention;

[0023] Figure 9 This is a schematic diagram illustrating the grasping of various types of easily slippery objects based on tactile feedback according to the present invention;

[0024] Figure 10 This is a schematic diagram illustrating the complete implementation process of the visual-tactile fusion secure grasping method of the present invention;

[0025] Figure 11 This is a flowchart of the visual-tactile fusion closed-loop secure grasping method of the present invention. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. These embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Without departing from the core idea of ​​the present invention, those skilled in the art can make equivalent substitutions or adaptive adjustments to the vision model, grasping model, sensor type, robotic arm model, and gripper structure.

[0027] like Figure 1 As shown, the system of this invention mainly consists of a vision acquisition module, a target segmentation module, a work area pose estimation module, a grasping posture prediction module, a coordinate transformation module, a robot execution module, a tactile perception module, and a safety control module. The vision acquisition module provides the target segmentation module and the work area pose estimation module with images and depth information of the work scene; the processing results of the target segmentation module and the work area pose estimation module jointly provide input for grasping posture prediction; after the grasping posture prediction module outputs candidate grasping postures, the coordinate transformation module converts them into robot-executable postures; the robot execution module performs approach, gripping, lifting, transporting, and placement actions; the tactile perception module feeds back the contact state between the gripper and the ceramic plate target to the safety control module; the safety control module outputs control commands to the robot execution module based on the tactile feedback and the gripper state.

[0028] like Figure 2 As shown, in one specific embodiment, the system mainly consists of an RGB-D camera, a robotic arm, a two-finger gripper, a tactile sensor, a vision computing unit, a grasping posture prediction unit, and a robot control unit. The RGB-D camera is used to acquire color and depth images of the work scene; the robotic arm is used to perform approach, descent, grasping, lifting, transporting, and placement actions; the two-finger gripper is installed at the end of the robotic arm for gripping the ceramic plate target; the tactile sensor is located at the gripper's contact surface to detect the contact state and force changes between the gripper and the ceramic plate. The vision computing unit is used to perform target segmentation, work area pose estimation, and grasping posture generation; the robot control unit is used to control the robotic arm and gripper movements based on visual results and tactile feedback.

[0029] Before the system operates, the camera, robotic arm, and gripper are initialized, and camera intrinsic parameters, hand-eye calibration matrix, robot motion parameters, gripper opening and closing parameters, tactile sensor parameters, and safety threshold parameters are loaded. The safety threshold parameters include target segmentation confidence threshold, target mask minimum area threshold, gripping score threshold, gripper maximum opening width, robotic arm movement speed and acceleration, tactile contact point number threshold, tactile contact area threshold, tactile resultant force threshold, stable contact duration frames, and placement / release height.

[0030] After system startup, the RGB-D camera acquires images of the work scene. The target segmentation module performs instance segmentation on the porcelain plate targets in the images, obtaining the target's mask, bounding box, category confidence score, and target region depth information. Targets with confidence scores below a preset threshold, mask areas smaller than a preset area, or invalid depth information are removed by the system to prevent false detections from entering the grasping process. For porcelain plate targets that meet the criteria, the system extracts local point clouds or candidate regions based on their mask areas and depth images, which serve as input for subsequent grasping pose prediction. The instance segmentation model can employ the YOLO11 instance segmentation network or a deep learning model with equivalent target segmentation capabilities; this invention is not limited to a specific instance segmentation model. Figure 3 The results of the porcelain plate segmentation are shown. Figure 4 The results of the work area identification and segmentation are shown.

[0031] Simultaneously, the system utilizes a six-degree-of-freedom pose estimation method to estimate the pose of the work area or reference object. The reference object can be a sandbox, pallet, calibration plate, work frame, or other reference object with a stable spatial position. Based on the pose of the reference object relative to the camera coordinate system and the calibration relationship between the camera coordinate system and the robot's base coordinate system, the system establishes a work world coordinate system. Through this work world coordinate system, the system can uniformly represent the target position, the robotic arm's motion pose, and the placement area position, thereby reducing the impact of transformation errors between different coordinate systems on the grasping action. Figure 5 The process of pose estimation and coordinate reference establishment in the work area is shown.

[0032] After obtaining the coordinate references of the target area and the working area of ​​the ceramic plate, the system generates multiple candidate grasping poses through the grasping pose prediction module. Each candidate grasping pose includes the grasping center position, the gripper approach direction, the gripper posture, the gripper opening width, and the grasping score. The grasping pose prediction model can use the GraspNet six-degree-of-freedom grasping pose prediction network, or a model with equivalent candidate grasping pose generation capabilities. The system filters candidate grasping poses, prioritizing those with higher grasping scores, gripper opening widths that meet the gripper's execution capabilities, grasping points located within the target mask area, grasping depths consistent with the target area depth, and reachable by the robotic arm. Candidate grasping poses that do not meet the safety conditions are discarded by the system. If no grasping pose meets the conditions, the system re-acquires images or returns to the grasping failure status. Figure 6 The process of generating and filtering candidate grabbing poses is shown.

[0033] After determining the final grasping pose, the system converts the grasping pose in the camera coordinate system to the execution pose in the robot base coordinate system. Combining this with the tool coordinate offset between the gripper center and the robot flange, the system calculates the actual pose the robotic arm's end effector should reach. The robotic arm first moves to the pre-grasping position above the target, positioning the gripper at a safe height above the ceramic disc target and aligning the gripper's attitude with the target grasping direction or the work area direction. Subsequently, the robotic arm slowly descends along a preset approach direction. During the descent, the system limits the robotic arm's speed and acceleration to prevent the end effector from impacting the ceramic disc or surrounding support media. Figure 7 The image shows the robotic arm approaching and gripping at low speed.

[0034] When the robotic arm reaches the vicinity of the gripping position, the gripper begins to close at a preset low speed, and tactile sensors simultaneously collect tactile data from the gripper's contact surface. Based on this tactile data, the system calculates the number of effective contact points, the connectivity of the contact area, the total contact force, and the duration of continuous stable contact. When the number of effective contact points reaches a preset threshold, the contact points form a stable connected area, the total contact force meets the safety gripping requirements, and these conditions remain stable for several consecutive frames, the system determines that stable contact has been established between the gripper and the ceramic disc. If stable contact is not detected after the gripper has closed to a preset stroke or after a preset time, the system stops the current gripping action and either attempts to re-grip or safely exits the gripper.

[0035] After detecting stable contact, the system enters the clamping confirmation phase. The grippers perform small closing or holding movements within a safe range to confirm that the ceramic disc is in a stable clamping state. At this time, the system continues to read tactile data. If the tactile contact force increases abnormally, further closing is stopped to avoid excessive clamping force that could damage the ceramic disc. If the tactile contact force decreases significantly or the contact area shrinks, the system is judged to be clamping unstable and a re-clamping or exit strategy is executed.

[0036] After successful gripping confirmation, the system performs a post-grip reinforcement operation. The grippers make one or more fine adjustments at low speed and with short strokes, maintaining a brief period of stability after each adjustment, and then re-checking the tactile contact state and changes in the gripper opening. This process eliminates minor gaps in the initial gripping, improves the stability of the ceramic disc in the grippers, and avoids excessive localized stress caused by a large, sudden closure.

[0037] Subsequently, the robotic arm enters the lifting phase. To reduce the risk of the ceramic disc slipping or falling, the system first performs a low-height, low-speed initial lifting phase. During this phase, the system continuously monitors changes in the gripper opening and the tactile contact status. If an abnormal increase in the gripper opening, a significant decrease in tactile contact force, or a significant reduction in the contact area is detected, a risk of slippage is identified, and the robotic arm immediately stops lifting and performs re-reinforcement, re-grip, or safe return based on the current status. If the first-phase lifting verification is successful, the robotic arm continues to lift the ceramic disc to a safe handling height and moves it above the preset placement area. Figure 8 It demonstrates the contact status and slip risk determination based on tactile data.

[0038] During the placement phase, the robotic arm first moves to a safe height above the placement area, then descends at a low speed to the preset release height. Upon reaching the release height, the grippers open at a preset release speed, allowing the ceramic disc to smoothly detach from the grippers and be placed at the target position. After release, the robotic arm withdraws in a safe direction to the preset height. If the tactile sensor still detects abnormal contact between the grippers and the ceramic disc during release, the system can delay withdrawal or continue opening the grippers to avoid the grippers dragging the ceramic disc and causing secondary damage.

[0039] Figure 9 This diagram illustrates the grasping of various types of easily slippery objects based on tactile feedback. While the diagram serves to demonstrate how the tactile sensing module and safety control strategy can provide contact feedback and gripping stability assessment under different object contact states, the primary focus of this invention remains the safe grasping and placement of fragile ceramic discs. Figure 10 The complete implementation process of visual-tactile fusion secure gripping is shown, including free posture, work area recognition, moving the work area, hand-eye camera recognition of the ceramic plate, obtaining the gripping posture, and mechanical gripper gripping. Figure 11 The overall process of the visual-tactile fusion closed-loop secure grasping method of the present invention is shown.

[0040] The system consists of six parts, each corresponding to a closed-loop security control system for the pre-grabbing, during-grabbing, and post-grabbing stages.

[0041] 1. Visual candidate quality score and depth consistency screening

[0042] For each candidate target mask Computational visual quality score:

[0043]

[0044] in For category confidence, For the mask area, The effective depth ratio of the mask, Penalty for mask overlap; For area normalization parameters, , , , These are the weighting coefficients for class confidence, mask area, depth effectiveness, and overlap penalty, respectively. Combined with the in-mask depth robust variance... :

[0045]

[0046] in For the first The depth consistency determination result of each candidate target The set of effective depth points within the candidate target mask region. The number of effective depth points, The minimum effective depth point threshold. The depth fluctuation threshold; The square brackets indicate the logical "AND" condition.

[0047] Used to filter stable and reliable crawling targets.

[0048] 2. Consistency judgment of captured point projection

[0049] Candidate crawl center Projected onto image:

[0050]

[0051] in The projection scale factor. and The pixel coordinates of the candidate capture center projected onto the image plane. For the camera intrinsic parameter matrix, The three-dimensional coordinates of the candidate capture center in the camera coordinate system.

[0052] And determine whether it falls within the mask area and whether the depth error meets the threshold:

[0053]

[0054] in For the first Projection consistency determination results of candidate crawling points This indicates whether the candidate grab center projection point is located within the mask area. The depth error of the candidate grab center. The allowable depth error threshold; It represents the logical "AND".

[0055] 3. Combine crawling score and candidate re-ranking

[0056] Overall candidate capture score:

[0057]

[0058] in To capture a comprehensive score for candidates, The raw grab score is used to represent the candidate grab pose. To capture the consistency determination result of the center mask, To capture the center depth error, The angle between the approach direction of the grippers and the desired gripping direction. This refers to the gripper opening width, collision, or boundary crossing penalty. , , , , These are the weights of the corresponding scoring items.

[0059] The top-k candidate capture poses are reordered to achieve safe-priority capture.

[0060] 4. Stable contact discrimination based on tactile array

[0061] Set of tactile activation points for each finger:

[0062]

[0063] in For fingers The set of activated tactile units on the upper surface, For the first in the tactile array Each tactile unit For fingers Upper The normal contact force detected by each tactile unit The normal force threshold at which the tactile unit determines effective contact. and These represent the two fingers of the gripper.

[0064] Maximum connected contact block:

[0065]

[0066] in For fingers Maximum connected contact block size Set of tactile activation points Connected subsets in This represents the number of tactile units contained in the connected subset. This is an adjacency diagram of the tactile array. express In the adjacency graph Upward connected.

[0067] Combined total force And continuous stable frames to determine whether the clamping has formed a stable contact.

[0068] in For fingers The total contact force, continuous stable frames indicate that the tactile contact state meets the stable contact condition in multiple consecutive frame sampling, which is used to reduce misjudgment caused by single frame noise.

[0069] 5. Initial Contact and Candidate Hierarchy Determination

[0070] Define initial contact:

[0071]

[0072] in This is the initial contact determination result. This represents the number of activation points in the haptic array that satisfy the contact threshold. The minimum number of activation points required for initial contact. The normal response intensity or resultant normal force detected by the haptic array. The minimum normal response threshold required for initial contact. It represents the logical "OR".

[0073] Two-finger standard grab candidate:

[0074]

[0075] in To standardize the capture of candidate judgment results, and The results of stable contact determination for the two fingers are as follows: The threshold number of activation points required for standard crawling. The normal response threshold required for standard capture; It represents the logical "AND".

[0076] Special criteria for small objects:

[0077]

[0078] in Retrieve candidate judgment results for small objects or objects with small contact areas. and These are the upper and lower limits for the number of activation points, respectively. For the corresponding normal response threshold, The resultant force detected by the two-finger tactile sensor. This corresponds to the resultant force threshold. Indicates the logical "and". It represents the logical "OR".

[0079] Final crawl candidates:

[0080]

[0081] in To ultimately capture the candidate judgment results, To standardize the capture of candidate judgment results, Capture candidate judgment results for small objects or small contact areas; It represents the logical "OR".

[0082] 6. Tactile slip discrimination and phased lifting verification during the lifting phase

[0083] During the lifting process, the baseline state Compared with the current state Comparison: Among them This serves as the baseline tactile state before the lifting process begins. The two measurements represent the current tactile state during the lifting process, and are used to compare whether there are significant changes in the contact area, contact force, and contact position.

[0084]

[0085] in For fingers The slip determination result, This represents the maximum change in the connected contact block. This represents the change in the number of tactile activation points. The current total contact force is maintained relative to the reference total contact force. To maintain the current normal response relative to the reference normal response, This represents the offset of the tactile contact center. , , , , These are the corresponding judgment thresholds. It represents the logical "OR".

[0086] Two-finger sliding detection:

[0087]

[0088] in The result is the overall slippage determination of the gripper. and The results show the sliding judgment of two fingers respectively; It represents the logical "OR".

[0089] Combined with the verification of the first stage of low-altitude, low-speed lifting:

[0090]

[0091] in To verify the results during the lifting phase, Indicates a state of no contact force. Indicates the slip state. This indicates the lifting height or end displacement during the lifting phase. This represents the minimum effective lift height threshold; Represents the logical "NOT". It represents the logical "AND".

[0092] Ensure that unstable clamping and slippage risks are identified before lifting, forming a closed-loop safety control.

[0093] Through the above implementation methods, this invention achieves complete closed-loop control from porcelain plate target recognition, work area positioning, grasping posture generation, coordinate transformation, robotic arm approach, tactile contact confirmation, clamping and reinforcement, staged lifting, to safe placement. This implementation method can reduce the risk of porcelain plate damage while ensuring grasping stability, and is especially suitable for robotic grasping and transfer tasks of fragile targets such as delicate porcelain plates, ancient ceramics, and thin-walled vessels.

Claims

1. A visual-tactile fusion closed-loop secure grasping method for fragile porcelain plates, characterized in that, The process includes the following steps: S1. Acquire color and depth images of the work scene through a vision acquisition module, and synchronize the color and depth images to obtain RGB-D image data containing the porcelain plate target to be captured and the work area; S2. Perform instance segmentation on the porcelain plate target based on the RGB-D image data to obtain the target mask, bounding box, category confidence, and target area depth information of the porcelain plate target, and filter the segmentation results according to the category confidence, target mask area, and depth effectiveness to obtain porcelain plate target areas that can enter the capture process; S3. Perform six-degree-of-freedom pose estimation on the work area or work reference object, and establish a coordinate reference for the work area in combination with the calibration relationship between the camera and the robot, so that the target area of ​​the ceramic plate, the motion pose of the robotic arm and the position of the placement area can be represented under a unified coordinate relationship. S4. Generate a local point cloud or grasping input data based on the target area of ​​the ceramic plate and its depth information, and generate multiple candidate grasping poses through the grasping posture prediction module. Each candidate grasping pose includes at least the grasping position, grasping direction, gripper posture, gripper opening width, and grasping score. S5. Filter the multiple candidate grasping poses according to the grasping score, gripper opening width, target mask constraint, depth consistency, grasping direction constraint, and robotic arm reachability constraint to determine the final grasping pose, and convert the final grasping pose from the camera coordinate system to the execution pose in the robot base coordinate system. S6. Control the robotic arm to move to the pre-grasping position corresponding to the execution pose, and then slowly descend to the grasping position along a preset safe approach direction. Control the two-finger gripper to close at a preset low speed, while the tactile sensing module collects tactile data between the gripper and the ceramic plate target. S7. Based on the number of effective contact points, contact area connectivity, total contact force, and continuous stabilization time in the tactile data, determine whether a stable contact has been formed between the gripper and the ceramic disc target. If a stable contact has not been formed, stop the current gripping action and proceed to the re-grip or safe exit process. S8. After determining that a stable contact has been formed, control the gripper to perform gripping confirmation and post-grip reinforcement within a preset safety range, and continuously judge whether there is gripping instability or excessive gripping force based on changes in tactile force and gripper opening status. S9. Control the robotic arm to perform a first-stage lifting at a low height and low speed, and judge whether there is a risk of slippage based on changes in gripper opening, tactile contact force, and contact area during the lifting process. If there is a risk of slippage, stop the lifting and perform re-reinforcement, re-grip, or safe return. S10. After the first stage of lifting verification is passed, control the robotic arm to move the ceramic plate target to the placement area, and complete the safe placement by slow descent, gripper release and robotic arm withdrawal.

2. The visual-tactile fusion closed-loop secure gripping method for fragile ceramic plates according to claim 1, characterized in that, In step S2, when performing instance segmentation on the porcelain plate target, an instance segmentation model is used to obtain the pixel-level target mask of the porcelain plate target. Segmentation results with target confidence lower than a preset confidence threshold, target mask area smaller than a preset area threshold, or invalid target region depth information are discarded. Only porcelain plate target regions that meet the screening conditions are retained as input for subsequent grasping posture prediction.

3. The visual-tactile fusion closed-loop secure gripping method for fragile ceramic plates according to claim 1, characterized in that, In step S3, the work area or work reference includes at least one of a sandbox, a tray, a calibration plate, and a work frame; by estimating the spatial pose of the work area or work reference relative to the camera coordinate system, and combining the calibration relationship between the camera coordinate system and the robot base coordinate system, a work world coordinate system or a work area coordinate system is established to generate a coarse positioning pose of the robotic arm relative to the target or work area.

4. The visual-tactile fusion closed-loop secure gripping method for fragile ceramic plates according to claim 1, characterized in that, In S4 and S5, after the grasping posture prediction module generates a local point cloud of the target or grasping input data based on the target mask and depth information, it outputs multiple candidate grasping postures suitable for execution by the two-finger gripper. When the safety control module screens the candidate grasping postures, it excludes at least the candidate grasping postures where the gripper opening width exceeds the maximum opening capacity of the gripper, the grasping point deviates from the target mask area, the grasping depth is inconsistent with the depth of the target area, the grasping direction does not meet the safety access requirements, or the robotic arm cannot reach the candidate grasping posture.

5. The visual-tactile fusion closed-loop secure gripping method for fragile ceramic plates according to claim 1, characterized in that, In step S6, after converting the final grasping pose into the execution pose in the robot base coordinate system, the tool coordinate offset between the gripper center and the robot flange is also combined to calculate the actual pose that the end effector should reach. The robot arm limits its speed and acceleration as it approaches the ceramic plate target to reduce the risk of the end effector impacting the ceramic plate target or the surrounding supporting medium.

6. The visual-tactile fusion closed-loop secure gripping method for fragile ceramic plates according to claim 1, characterized in that, In step S7, when the number of effective contact points reaches a preset contact point number threshold, the contact area forms a stable connected distribution, the total contact force reaches a preset safe clamping requirement, and the above states remain stable within several consecutive frames, it is determined that a stable contact has been formed between the gripper and the ceramic disk target; when the gripper closes to a preset stroke or fails to form a stable contact after a preset time, the gripper closure is stopped and the process of re-grabbing or safe exit is initiated.

7. The visual-tactile fusion closed-loop secure gripping method for fragile ceramic plates according to claim 1, characterized in that, In step S8, after a stable contact is formed, the gripper performs one or more fine adjustments at a low speed and with a small stroke. After each fine adjustment, it maintains a preset stable time and detects the tactile contact state and the change in the gripper opening again. When an abnormal increase in tactile contact force is detected, further closing is stopped. When a significant decrease in tactile contact force or a reduction in the contact area is detected, the gripping state is determined to be unstable.

8. The visual-tactile fusion closed-loop secure gripping method for fragile ceramic plates according to claim 1, characterized in that, In S9, the lifting height and speed of the first stage of lifting are less than the lifting height and speed of the subsequent safe handling stage. During the first stage of lifting, if an abnormal increase in the gripper opening, a significant decrease in tactile contact force, or a significant reduction in the contact area is detected, it is determined that there is a risk of slippage, and the robotic arm is controlled to stop lifting. In S10, the robotic arm first moves to a safe height above the placement area, and then descends at a low speed to a preset release height. The gripper opens at a preset release speed, allowing the ceramic disc target to detach from the gripper and be placed in the placement area. After release, the robotic arm withdraws to a preset height in a safe direction. If the tactile sensing module still detects abnormal contact during the release process, the withdrawal is delayed or the gripper continues to open.

9. The visual-tactile fusion closed-loop secure gripping method for fragile ceramic plates according to claim 1, characterized in that, The closed-loop safety control process for each stage before, during, and after the ceramic plate is grasped is as follows: S1. Visual candidate quality score and depth consistency screening; For each candidate target mask Computational visual quality score: ； in For category confidence, For the mask area, The effective depth ratio of the mask, Penalty for mask overlap; For area normalization parameters, , , , These are the weighting coefficients for class confidence, mask area, depth effectiveness, and overlap penalty, respectively; combined with the robust variance of depth within the mask. : ； in For the first The depth consistency determination result of each candidate target The set of effective depth points within the candidate target mask region. The number of effective depth points, The minimum effective depth point threshold. The depth fluctuation threshold; The square brackets indicate a logical AND; Used to filter stable and reliable crawling targets; S2. Consistency judgment of capture point projection; Candidate crawl center Projected onto image: ； in The projection scale factor. and The pixel coordinates of the candidate capture center projected onto the image plane. For the camera intrinsic parameter matrix, The three-dimensional coordinates of the candidate capture center in the camera coordinate system; And determine whether it falls within the mask area and whether the depth error meets the threshold: ； in For the first Projection consistency determination results of candidate crawling points This indicates whether the candidate grab center projection point is located within the mask area. The depth error of the candidate grab center. The allowable depth error threshold; This represents the logical AND; S3. Combine crawling scores with candidate re-ranking; Overall candidate capture score: ； in To capture a comprehensive score for candidates, The raw grab score is used to represent the candidate grab pose. To capture the consistency determination result of the center mask, To capture the center depth error, The angle between the approach direction of the grippers and the desired gripping direction. This refers to the gripper opening width, collision, or boundary crossing penalty. , , , , These are the weights of the corresponding scoring items; Reorder the top-k candidate capture poses to achieve safe-priority capture; S4. Stable contact discrimination based on tactile array; Set of tactile activation points for each finger: ； in For fingers The set of activated tactile units on the upper surface, For the first in the tactile array Each tactile unit For fingers Upper The normal contact force detected by each tactile unit The normal force threshold at which the tactile unit determines effective contact. and These represent the two fingers of the gripper; Maximum connected contact block: ； in For fingers Maximum connected contact block size Set of tactile activation points Connected subsets in This represents the number of tactile units contained in the connected subset. This is an adjacency diagram of the tactile array. express In the adjacency graph Up-connected; Combined total force And continuous stable frames, to determine whether the clamping has formed a stable contact; in For fingers The total contact force, and the continuous stable frame indicates that the tactile contact state meets the stable contact condition in multiple consecutive frame samplings, which is used to reduce misjudgment caused by single frame noise; S5. Initial contact and candidate layering for grabbing; Define initial contact: ； in This is the initial contact determination result. This represents the number of activation points in the haptic array that satisfy the contact threshold. The minimum number of activation points required for initial contact. The normal response intensity or resultant normal force detected by the haptic array. The minimum normal response threshold required for initial contact. This represents the logical "OR" operator. Two-finger standard grab candidate: ； in To standardize the capture of candidate judgment results, and The results of stable contact determination for the two fingers are as follows: The threshold number of activation points required for standard crawling. The normal response threshold required for standard capture; This represents the logical AND; Special criteria for small objects: ； in Retrieve candidate judgment results for small objects or objects with small contact areas. and These are the upper and lower limits for the number of activation points, respectively. For the corresponding normal response threshold, The resultant force detected by the two-finger tactile sensor. This corresponds to the resultant force threshold. The logical "AND" is represented by the symbol "AND". This represents the logical "OR" operator. Final crawl candidates: ； in To ultimately capture the candidate judgment results, To standardize the capture of candidate judgment results, Capture candidate judgment results for small objects or small contact areas; This represents the logical "OR" operator. S6. Tactile slip discrimination and phased lifting verification during the lifting phase; During the lifting process, the baseline state Compared with the current state Comparison: Among them This serves as the baseline tactile state before the lifting process begins. The two measurements represent the current tactile state during the lifting process, and are used to compare whether there are significant changes in the contact area, contact force, and contact position. ； in For fingers The slip determination result, This represents the maximum change in the connected contact block. This represents the change in the number of tactile activation points. The current total contact force is maintained relative to the reference total contact force. To maintain the current normal response relative to the reference normal response, This represents the offset of the tactile contact center. , , , , These are the corresponding judgment thresholds. This represents the logical "OR" operator. Two-finger sliding detection: ； in The result is the overall slippage determination of the gripper. and The results show the sliding judgment of two fingers respectively; This represents the logical "OR" operator. Combined with the verification of the first stage of low-altitude, low-speed lifting: ； in To verify the results during the lifting phase, Indicates a state of no contact force. Indicates the slip state. This indicates the lifting height or end displacement during the lifting phase. This represents the minimum effective lift height threshold; Represents the logical "NOT". This represents the logical AND; Ensure that unstable clamping and slippage risks are identified before lifting, forming a closed-loop safety control.

10. A visual-tactile fusion closed-loop secure gripping system for fragile porcelain plates, characterized in that, The system is used to perform the method according to any one of claims 1 to 9, the system comprising a vision acquisition module, a target segmentation module, a work area pose estimation module, a grasping posture prediction module, a coordinate transformation module, a robot execution module, a tactile perception module, and a safety control module; the vision acquisition module is used to acquire color images and depth images of the work scene; the target segmentation module is used to identify and extract the target area of ​​the ceramic plate; the work area pose estimation module is used to acquire the six-degree-of-freedom pose of the work area or work reference object and establish a work coordinate reference; the grasping posture prediction module is used to generate candidate grasping poses based on the target area of ​​the ceramic plate and depth information; the coordinate transformation module is used to convert the grasping pose in the camera coordinate system into the execution pose in the robot base coordinate system; the robot execution module includes a robotic arm and a two-finger gripper, used to perform approach, gripping, lifting, carrying, and placing actions; the tactile perception module is used to acquire data related to the contact state, force distribution, and slippage risk between the gripper and the ceramic plate target; The safety control module is used to perform closed-loop control of the gripper closure, gripping confirmation, post-grab reinforcement, first-stage lifting verification, and safe placement process based on visual results, tactile data, and gripper status.

Images