An intelligent robot arm grabbing method and system based on EIT tactile and monocular vision fusion

Abstract

This invention relates to the field of robot control and tactile perception technology, specifically to a robotic arm intelligent grasping method and system based on EIT tactile sensing and monocular vision fusion. The system acquires the position, material, and prior grasping parameters of the target object through monocular vision, and guides the robotic arm to the target grasping pose through coordinate mapping. During contact interaction, boundary voltage signals are collected based on the EIT flexible tactile sensing unit, and after differential pressure quantization and impedance distribution reconstruction, global contact pressure and two-dimensional contact pressure distribution images are obtained. Multi-dimensional tactile features such as contact area, peak pressure, and pressure center are extracted from these images and used as feedback input to the closed-loop control module to adaptively generate gripper opening and closing control quantities until stable grasping conditions are met. This invention can improve the stability, precision, and safety of the robotic arm's grasping of targets of different materials, and reduce the probability of damage and slippage caused by local stress concentration or insufficient gripping force.

Description

Technical Field

[0001] This invention relates to the field of robot control and tactile perception technology, specifically to a robotic arm intelligent grasping method and system based on EIT tactile and monocular vision fusion. Background Technology

[0002] With the rapid development of industrial automation and artificial intelligence technologies, the application scope of robotic arms in unstructured environments is expanding, and they have been widely penetrated into diverse scenarios such as intelligent manufacturing, warehousing and logistics, and medical services. In these complex environments, robotic arms no longer simply perform preset repetitive actions, but need to frequently handle objects with unknown materials, shapes, weights, and physical stiffness. However, existing robotic arm grasping solutions still exhibit significant limitations in robustness, versatility, and precision when facing such highly uncertain tasks.

[0003] Currently, mainstream robotic arm grasping systems heavily rely on vision guidance. These methods primarily acquire two-dimensional or three-dimensional images of the target object using visual sensors such as cameras, and then use visual algorithms to calculate the target's geometric center, spatial pose, and grasping point. However, pure vision solutions are essentially position-based closed-loop or open-loop control strategies, inherently possessing perceptual blind spots: during the transient state of contact between the gripper and the object, and in the subsequent steady-state grasping phase, the vision system cannot penetrate the contact surface to obtain true physical interaction information. This makes it difficult for the system to perceive the magnitude, direction, and dynamic distribution changes of the contact force. For fragile items such as glassware, easily deformable soft objects such as fruit, or objects with smooth surfaces, single-vision guidance is highly susceptible to irreversible mechanical damage due to excessive grasping force, or to object slippage or even grasping failure due to insufficient grasping force.

[0004] To compensate for the lack of visual information, researchers introduced traditional force / tactile sensors. While traditional wrist force / torque sensors can measure the overall six-dimensional resultant force, they cannot reflect the fine pressure characteristics of the contact surface. Traditional tactile solutions based on piezoresistive and capacitive arrays often face problems such as limited spatial resolution, small coverage area, complex wiring, and poor flexibility. Under complex contact patterns, these sensors can only be used for simple contact detection or threshold judgment, and are unable to capture high-dimensional physical features such as local stress concentration and contact area evolution. They cannot serve as the core feedback quantity for fine adjustment of grasping force, resulting in visual-tactile collaboration remaining at a shallow state estimation level and lacking a deep physical closed loop.

[0005] Existing applications for visual-tactile fusion (such as CN120715896A, CN117207190B, and CN121245810A) mostly focus on semantic-level recognition or robot pose state estimation, without in-depth modeling of the physical properties of the object material (such as stiffness and friction coefficient). This makes it difficult to ensure rapid response while also meeting the requirements for smooth grasping of soft / hard and light / heavy objects, resulting in insufficient robustness of the control strategy.

[0006] In summary, to achieve stable and precise grasping of unknown objects in unstructured environments, a novel technical solution is urgently needed. This solution should fully leverage the high-resolution imaging potential of EIT tactile sensors, utilize EIT imaging algorithms to reconstruct the conductivity variation distribution of the contact area and extract high-dimensional contact physical features, and combine adaptive control theory to address the problems of reconstruction delay, single feedback dimension, and insufficient algorithm adaptability. Summary of the Invention

[0007] This invention aims to address the problem in existing technologies where robotic arms suffer from poor adaptability and susceptibility to object damage due to the lack of precise, high-dimensional real-time force feedback during grasping tasks. Specifically, this invention provides a method and system capable of obtaining global and two-dimensional contact pressure distribution images based on EIT tactile feedback, extracting multi-dimensional tactile features from them, and ultimately achieving stable and appropriate grasping of objects of different materials through closed-loop control.

[0008] To achieve the above objectives, this invention provides a robotic arm intelligent grasping method based on the fusion of EIT tactile feedback and monocular vision, comprising the following steps:

[0009] S1: Visual Prior Acquisition: Scene images are acquired using a monocular vision sensing unit to identify the target position and material type of the target object to be grasped, and the target grasping pressure threshold and target contact area threshold are determined based on the material type. The target grasping pressure threshold and target contact area threshold serve as initial reference values ​​for subsequent closed-loop force control steps and are dynamically adjusted based on EIT tactile feedback during actual contact.

[0010] S2: Visual guidance positioning: Based on the target position, the target physical coordinates of the target object in the base coordinate system of the robotic arm are obtained through coordinate transformation, and the end gripper of the robotic arm is driven to move to the hovering pose and the corresponding grasping pose. The coordinate transformation is used to establish the planar position correspondence between the image coordinate system and the working plane. The movement of the robotic arm along the Z-axis is controlled by the preset hovering height and grasping height.

[0011] S3: EIT tactile sensing and pressure quantization: During the contact process between the gripper and the target object, the boundary voltage signal of the EIT flexible tactile sensing unit set on the contact surface or inner wall area of ​​the gripper is collected in real time. The global contact pressure is obtained based on the difference between the real-time boundary voltage and the no-load reference voltage, and a two-dimensional contact pressure distribution image is reconstructed.

[0012] S4: Multidimensional tactile feature extraction: Process the two-dimensional contact pressure distribution image to extract multidimensional tactile features including at least contact area, peak pressure, pressure center and global contact pressure;

[0013] S5: Closed-loop force control: Generate gripper opening and closing control quantities based on the target gripping pressure threshold, real-time global contact pressure and the multi-dimensional tactile features, and drive the gripper to perform closing, releasing or holding actions until the preset stable gripping conditions are met.

[0014] Further, step S1 includes:

[0015] The scene image is input into a visual recognition model to obtain the target bounding box, material category, and recognition confidence of the candidate target.

[0016] The target to be captured is determined based on the recognition confidence level, and the center coordinates of the target pixels are calculated from its target bounding box.

[0017] Based on the material category, a preset material-grip parameter mapping table is queried to determine the target gripping pressure threshold and the target contact area threshold;

[0018] If no target object is identified within the preset time window, the most recent valid identification result is retained as the current target prior.

[0019] If the target object is not identified within the preset time window, the current grasping task is stopped or the visual recognition is re-executed.

[0020] Further, step S2 includes:

[0021] Multiple calibration points are pre-selected, and their pixel coordinates in the image coordinate system and their physical coordinates in the robot arm base coordinate system are recorded respectively. The coordinate transformation matrix is ​​then obtained based on this.

[0022] The target pixel center coordinates are expanded to homogeneous coordinates, and then normalized after transformation by the coordinate transformation matrix to obtain the target physical coordinates of the target object in the robot arm base coordinate system.

[0023] The hovering pose and grasping pose of the robotic arm end effector are generated based on the target physical coordinates. When the pose error between the actual pose and the target pose is less than a preset threshold, it is determined that the robotic arm end effector has reached the corresponding target position.

[0024] Further, step S3 includes:

[0025] The EIT flexible tactile sensing unit includes a flexible conductive sensitive layer and multiple electrodes disposed at the edge of the flexible conductive sensitive layer;

[0026] The no-load reference voltage vector is acquired in a non-contact state, and the real-time boundary voltage vector is acquired during the grasping process. The differential voltage vector is calculated, and the global contact pressure is obtained based on the differential voltage vector.

[0027]

[0028] in, This represents the pressure quantization function obtained from calibration. It represents the difference between the real-time boundary voltage vector and the no-load reference voltage vector, that is, the differential voltage vector caused by the contact of the target object;

[0029] The global contact pressure is expressed as:

[0030]

[0031] in, Indicates the pressure calibration factor; The L2 norm of the differential voltage vector; Indicates the upper limit of pressure;

[0032] Based on the differential voltage vector, an inverse EIT problem is constructed, and the conductivity variation distribution is solved using a regularized objective function:

[0033]

[0034] in, Represents the reconstruction objective function; Represents the regularization coefficient; Represents the regularization term. This represents the pre-calculated sensitivity matrix; This represents the distribution of conductivity variation within the flexible conductive sensitive layer.

[0035] The obtained conductivity variation distribution is mapped to a two-dimensional contact pressure distribution image:

[0036]

[0037] in, This represents a two-dimensional contact pressure distribution image. This represents the pressure mapping function obtained through pre-calibration.

[0038] Furthermore, in S3, the EIT image reconstruction uses a regularization method to solve for the conductivity variation distribution. The regularization algorithm is one of the following: Tikhonov regularization algorithm, Landweber iterative algorithm, conjugate gradient least squares algorithm, or linear back projection algorithm.

[0039] Further, step S4 includes:

[0040] Threshold segmentation is performed on the two-dimensional contact pressure distribution image to obtain the effective contact area:

[0041]

[0042] in, Indicates the effective contact area; The horizontal index representing the pixel position in a two-dimensional contact pressure distribution image; The vertical index representing the pixel position in a two-dimensional contact pressure distribution image; Indicates position Pressure response value at the location; Indicates the contact area segmentation threshold;

[0043] Calculate the contact area and peak pressure based on the effective contact area:

[0044]

[0045]

[0046] in, Indicates the contact area; This indicates the number of pixels in the effective contact area. This indicates the peak pressure within the effective contact area;

[0047] Calculate the pressure center based on the pressure response distribution within the effective contact area:

[0048]

[0049]

[0050] and These represent the horizontal and vertical coordinates of the pressure center in the two-dimensional contact pressure distribution image, respectively.

[0051] The contact area, peak pressure, pressure center, and global contact pressure are used to construct a multi-dimensional tactile feature vector:

[0052]

[0053] in, This represents the multidimensional tactile feature vector at the current moment; Indicates the current contact area; Indicates the current peak pressure; and These represent the horizontal and vertical coordinates of the current pressure center, respectively. This represents the real-time global contact pressure obtained from the EIT pressure quantization step.

[0054] Further, step S5 includes:

[0055] The pressure error is calculated based on the target grasping pressure threshold and the real-time global contact pressure.

[0056]

[0057] in, Indicates the current pressure error; This indicates the target gripping pressure threshold determined based on the material of the target object; This represents the real-time global contact pressure obtained from the EIT tactile sensing step;

[0058] Based on the pressure error and the multidimensional tactile feature vector, a gripper opening and closing control value is generated:

[0059]

[0060] in, This indicates the gripper control quantity at the current moment; Represents the closed-loop control function; This represents the multidimensional tactile feature vector at the current moment;

[0061] Wherein, the closed-loop control function One of the following is employed: PID control, adaptive PID control, fuzzy PID control, impedance control, or model predictive control.

[0062] Furthermore, the closed-loop control function Using PID control, the gripper control quantity can be expressed as:

[0063]

[0064] in, , , These represent the proportional, integral, and derivative control coefficients, respectively, at least one of which can be adjusted based on the current multidimensional tactile feature vector.

[0065] Update the gripper opening degree based on the gripper control quantity u(t):

[0066]

[0067] in, This indicates the current gripper opening degree; This indicates the opening and closing degree of the grippers at the previous moment;

[0068] When the following conditions are met simultaneously, the gripper is determined to have formed a stable grip, and the current gripper opening degree is maintained, while the robotic arm is controlled to perform a lifting action:

[0069]

[0070]

[0071]

[0072] in, Indicates the allowable pressure error; Indicates the current contact area; Indicates the target contact area threshold; Indicates the current pressure center vector; This represents the pressure center vector at the previous moment; This indicates the threshold at which the pressure center can vary.

[0073] Furthermore, it also includes communication anomaly handling and degradation control steps:

[0074] During the execution of the capture task, the status of the communication link between the edge execution terminal and the computing server, as well as the status of the EIT haptic data, are monitored.

[0075] When the communication link is normal and the EIT haptic data is valid, closed-loop force control based on EIT haptic feedback is executed.

[0076] When the communication link is abnormal or the EIT haptic data is invalid, the closed-loop force control based on EIT haptic feedback is stopped and the preset safety degradation control strategy is switched.

[0077] The preset safety degradation control strategy includes one or more of the following: stopping the gripper from closing, maintaining the current gripper opening degree, releasing the target object, performing a preset safety action that limits displacement or force, or controlling the robotic arm to return to a safe position.

[0078] This invention also provides a robotic arm intelligent grasping system based on the fusion of EIT tactile feedback and monocular vision to implement the above method. The system includes:

[0079] Edge execution unit: Used to perform visual acquisition, target recognition, coordinate mapping, and robotic arm motion control, obtaining the target position and material type of the object to be grasped, and guiding the robotic arm's end effector gripper to the corresponding grasping pose of the target object. In one embodiment, the edge execution unit is deployed on the NVIDIA Jetson edge computing platform.

[0080] The computing server communicates with the edge execution terminal and is connected to the EIT flexible tactile sensing unit for performing EIT tactile data acquisition, pressure quantization, contact pressure image reconstruction, multi-dimensional tactile feature extraction, and closed-loop force control. In one embodiment, the computing server is deployed on a Windows general-purpose computing platform.

[0081] EIT Flexible Tactile Sensing Unit: Located on the contact surface or inner wall area of ​​the gripper at the end of the robotic arm, it includes a flexible conductive sensitive layer and multiple electrodes located at the edge of the flexible conductive sensitive layer, used to generate boundary voltage signals during the contact between the gripper and the target object.

[0082] Robotic arm and end effector: used to perform vision-guided positioning based on the target physical position output by the edge actuator, and to perform gripper opening and closing actions based on the gripper control quantity output by the computing server.

[0083] Communication link: used to transmit visual prior information, EIT processing results, gripper control commands and system status information between the edge execution end and the computing server.

[0084] Furthermore, the edge execution terminal includes a vision acquisition module, a vision recognition module, a coordinate mapping module, a robotic arm drive module, and a first communication module; the computing server includes an EIT data acquisition module, a pressure quantization module, an EIT imaging module, a tactile feature extraction module, a closed-loop control module, and a second communication module; the edge execution terminal and the computing server constitute a distributed collaborative control architecture.

[0085] Compared with the prior art, the present invention has the following significant advantages:

[0086] 1. This invention not only utilizes EIT differential voltage to obtain global contact pressure, but also reconstructs a two-dimensional contact pressure distribution image and extracts multi-dimensional tactile features such as contact area, peak pressure, and pressure center, providing richer contact state feedback for gripper control, enabling the robotic arm to perceive local stress concentration, contact area changes, and possible slippage trends.

[0087] 2. This invention combines the target grasping pressure threshold determined by visual prior with real-time EIT tactile feedback, and generates gripper control quantities based on pressure error and multi-dimensional tactile characteristics, so that the gripper can dynamically adjust the closing or releasing action according to the contact state of target objects of different materials, thereby achieving stable and non-excessive grasping.

[0088] 3. Peak pressure of the present invention It can be used to determine the risk of localized stress concentration or excessive compression, and the center of pressure. , Changes in the contact area can be used to determine whether a target object has a slipping tendency. It can be used to determine whether contact is sufficient. The above features work together in closed-loop control, which helps reduce the risk of fragile or deformable objects being crushed and improves the success rate of grasping in unstructured environments.

[0089] 4. This invention adopts a distributed architecture that coordinates edge execution and computing server. The edge execution is responsible for visual recognition, coordinate mapping, and robotic arm motion guidance, while the computing server is responsible for EIT tactile data processing, contact pressure image reconstruction, and closed-loop force control. This architecture allows for the replacement of visual sensing units, computing platforms, communication methods, or gripper actuators according to actual application scenarios, exhibiting good modularity and scalability.

[0090] 5. When the communication link is abnormal or the EIT tactile data is invalid, the present invention can switch to a preset security degradation control strategy to avoid the gripper from continuously applying force in the absence of reliable tactile feedback, thereby further improving the system's operational security. Attached Figure Description

[0091] Figure 1 This is a schematic diagram of the overall process of the adaptive grasping method of the robotic arm of the present invention;

[0092] Figure 2 This is a schematic diagram of the system architecture and data interaction connections of each module of the present invention;

[0093] Figure 3 This is a schematic diagram of the end gripper and flexible sensor in an embodiment of the present invention;

[0094] Figure 4 This is a diagram showing the arrangement of the electrode array around the flexible conductive sensitive layer in an embodiment of the present invention.

[0095] Figure 5 This is a schematic diagram of the EIT image reconstruction and multidimensional feature extraction process of the present invention;

[0096] Figure 6 This is a block diagram illustrating the working principle of the PID controller of the present invention;

[0097] Figure 7This is a flowchart illustrating the communication anomaly degradation and fault tolerance mechanism of the present invention.

[0098] In the figure, 1 is the end flange of the robotic arm, 2 is the monocular vision camera, 3 is the left gripper finger, 4 is the flexible conductive sensitive layer, 5 is the right gripper finger, and 6 is the gripper base. Detailed Implementation

[0099] To make the objectives, technical solutions, and beneficial effects of this invention clearer and more complete, the invention will be further described in detail below with reference to specific embodiments. The following embodiments are only used to illustrate the implementation of this invention and are not intended to limit the scope of protection of this invention.

[0100] This embodiment provides a robotic arm intelligent grasping method and system based on the fusion of EIT tactile sensing and monocular vision. The system adopts a distributed collaborative control architecture, including an edge execution end, a computing server, a monocular vision sensing unit, an EIT flexible tactile sensing unit, a robotic arm and an end effector gripper, and a communication link.

[0101] In this embodiment, the edge execution terminal is deployed on the NVIDIA Jetson edge computing platform for performing visual acquisition, target recognition, coordinate mapping, and robotic arm motion control; the computing server is deployed on the Windows general-purpose computing platform for performing EIT tactile data acquisition, pressure quantization, contact pressure image reconstruction, multi-dimensional tactile feature extraction, and closed-loop force control. It should be understood that the Jetson platform and Windows platform described above are merely exemplary implementations, and the edge execution terminal and computing server can also be implemented using other hardware or software platforms with corresponding computing and communication capabilities.

[0102] Before contact with the target object, the system acquires the target object's position and material priors through a monocular vision sensing unit and guides the robotic arm's end gripper to move to the grasping pose. After contact with the target object, the system acquires the boundary voltage changes during the contact process through an EIT flexible tactile sensing unit, thereby obtaining global contact pressure and two-dimensional contact pressure distribution images, and extracting multi-dimensional tactile features from them for closed-loop force control, thus achieving stable and adaptive grasping of target objects of different materials.

[0103] For detailed procedures, please refer to [link / reference]. Figure 1As shown, this method sequentially includes visual prior acquisition, visual guided localization, EIT tactile perception and pressure quantification, multi-dimensional tactile feature extraction, closed-loop force control, and stable gripping condition determination. The vision module obtains the target position, material type, and gripping threshold, and guides the robotic arm's end effector gripper to move to the gripping pose. The EIT flexible tactile sensing unit obtains global contact pressure and two-dimensional contact pressure distribution images, extracts the contact area, peak pressure, and pressure center, and uses these to adjust the gripper opening degree. When the stable gripping condition is not met, tactile feedback adjustment continues; when the condition is met, the gripper opening degree is maintained and a lifting action is performed.

[0104] Figure 1 The process within the diagram begins with the acquisition of the S1 visual prior. In this step, the monocular vision sensing unit acquires scene images and identifies the target pixel center and the target pixel center corresponding to the material category. , and material category The mapping relationship corresponding to the grabbing threshold is determined by the material-grabbing parameter mapping table. The material-grabbing parameter mapping table in the diagram is... The target grasping pressure threshold and the target contact area threshold are respectively and .

[0105] S2 vision-guided localization involves mapping target pixel coordinates to robotic arm base coordinates and planning the current pose, hovering pose, and grasping pose. The former outputs the target's physical coordinates. , The latter according to It sequentially passes through the current pose, hover pose, and grab pose.

[0106] Then, it proceeds to S3 EIT tactile sensing and pressure quantization, acquiring real-time boundary voltage vectors. And call the no-load reference ,according to Calculate the differential voltage and quantify the real-time global contact pressure. .

[0107] S4 Multidimensional Tactile Feature Extraction First Reconstructs a Two-Dimensional Contact Pressure Distribution Image Then extract the contact area, peak pressure, and pressure center, corresponding to , , and And construct a multi-dimensional tactile feature column vector at the current moment. The S5 closed-loop force control generates control quantities based on pressure error and multi-dimensional tactile characteristics, whereby... and in accordance with Generate control inputs, then update the gripper opening degree and issue control commands.

[0108] A stable crawling condition requires that three conditions be met simultaneously: , and When the judgment result is "no", the loop returns to S3 EIT tactile perception and pressure quantization; when the judgment result is "yes", the grasping and lifting process is entered, and the current gripper opening degree is maintained and the robotic arm is controlled to lift the target object and move it to the target position, and finally the process ends.

[0109] I. System Overall Architecture and Implementation Environment

[0110] This embodiment provides a robotic arm intelligent grasping system based on the fusion of EIT tactile sensing and monocular vision. The system includes an edge execution end, a computing server, an EIT flexible tactile sensing unit, a robotic arm and an end effector gripper, and a communication link. The system can be represented as follows:

[0111]

[0112] in, This refers to a robotic arm intelligent grasping system; Indicates the edge execution end; Indicates the computing server; This refers to the EIT flexible tactile sensing unit; This refers to the robotic arm and its end effector gripper. This represents the communication link between the edge execution end and the computing server.

[0113] Edge execution end This device is used for visual perception, target localization, and robotic arm motion control. The edge execution unit includes a vision acquisition module, a vision recognition module, a coordinate mapping module, a robotic arm drive module, and a first communication module. The vision acquisition module includes a monocular vision sensor unit for acquiring images of the target working area; the vision recognition module identifies the target position and material type of the target to be grasped; the coordinate mapping module converts the target object's position in the image coordinate system to its physical position in the robotic arm's base coordinate system; the robotic arm drive module controls the robotic arm's end effector to move to the grasping pose corresponding to the target object; and the first communication module interacts with the computing server for data exchange.

[0114] Computing server This system is used to complete EIT tactile data processing, pressure image reconstruction, multi-dimensional tactile feature extraction, and closed-loop force control. The computing server includes an EIT data acquisition module, a pressure quantization module, an EIT imaging module, a tactile feature extraction module, a closed-loop control module, and a second communication module. The EIT data acquisition module receives the boundary voltage signal output by the EIT flexible tactile sensing unit; the pressure quantization module calculates the global contact pressure based on the difference between the real-time boundary voltage and the no-load reference voltage; the EIT imaging module reconstructs the contact pressure distribution image based on boundary voltage changes; the tactile feature extraction module extracts the contact area, peak pressure, and pressure center from the contact pressure distribution image; the closed-loop control module generates gripper control quantities based on the target gripping pressure threshold and real-time tactile features; and the second communication module sends gripper control commands to the edge execution end.

[0115] For a detailed system architecture and data interaction connections between modules, please refer to [reference needed]. Figure 2 The system comprises an edge execution unit, a computing server, a communication link, and external hardware. The edge execution unit includes a vision acquisition module, a vision recognition module, a coordinate mapping module, a robotic arm drive module, and a first communication module, used to acquire scene images, visual prior information, target physical coordinates, and motion commands. The computing server includes an EIT data acquisition module, a pressure quantization module, an EIT imaging module, a tactile feature extraction module, a closed-loop control module, and a second communication module, used to complete boundary voltage acquisition, pressure calculation, image reconstruction, feature extraction, and gripper control quantity calculation. The external hardware includes a robotic arm and end effector gripper, an EIT flexible tactile sensing unit, and the target object to be grasped. The edge execution unit and the computing server interact via the communication link, exchanging visual prior information, tactile processing results, gripper control commands, and execution status, thereby forming a grasping control closed loop.

[0116] Figure 2 The description of this system generally includes edge execution terminals. Computing server Communication links External hardware. The edge execution unit and the computing server communicate bidirectionally via a communication link to exchange visual prior information, gripper control commands, and execution status.

[0117] The edge execution unit sequentially comprises a vision acquisition module, a vision recognition module, a coordinate mapping module, a robotic arm drive module, and a first communication module. The vision acquisition module includes a monocular camera outputting scene images; the vision recognition module outputs images through a target detection network and a material classifier. , and The coordinate mapping module performs perspective transformations and coordinate mappings and outputs the results. , The robotic arm drive module executes motion control and gripper control, issuing motion commands to reach... , , Different positions and gripper opening degrees .

[0118] The computing server is configured with an EIT data acquisition module, a pressure quantization module, an EIT imaging module, a tactile feature extraction module, a closed-loop control module, and a second communication module. The EIT data acquisition module performs frame parsing and outputs voltage data; the pressure quantization module performs differential and norm calculations and outputs... and The EIT imaging module solves and outputs the results through reciprocity recovery and inverse problem solving. and The tactile feature extraction module extracts the contact area, peak pressure, and pressure center, corresponding to... , , and The closed-loop control module utilizes error and characteristic feedback. , generate .

[0119] External hardware includes a robotic arm and an end effector gripper. The system consists of an EIT flexible tactile sensing unit and a target object to be grasped. The robotic arm and end effector receive pose and gripper control commands and output position, orientation, and status. The EIT flexible tactile sensing unit includes a flexible conductive sensitive layer and an edge multi-electrode array. The target object to be grasped has different shapes, materials, and positions and makes contact with the gripper. The target object exerts pressure on the EIT flexible tactile sensing unit, and the EIT flexible tactile sensing unit feeds back the contact status to the system.

[0120] The EIT flexible tactile sensing unit The EIT flexible tactile sensing unit is located on the contact surface or inner wall of the gripper at the end of the robotic arm to sense the contact state between the gripper and the target object. The EIT flexible tactile sensing unit includes a flexible conductive sensitive layer and multiple electrodes disposed at the edge of the flexible conductive sensitive layer. The flexible conductive sensitive layer can be made of conductive hydrogel, conductive silicone, or conductive composite elastomer. When the gripper contacts the target object, the flexible conductive sensitive layer deforms under pressure, generating a local impedance change, which is reflected by a boundary voltage signal.

[0121] For details on the structure of the end effector gripper and the flexible tactile sensing unit, please refer to [reference needed]. Figure 3The end flange 1 of the robotic arm is connected to the gripper base 6. The gripper base 6 has a left gripper finger 3 and a right gripper finger 5. The target object is located between the two gripper fingers. The left gripper finger 3 integrates an EIT flexible tactile sensing unit and an electrode array, while the right gripper finger 5 serves as the contact surface for maintaining gripping. The contact area of ​​the left gripper finger 3 is equipped with an EIT flexible tactile sensing unit, which includes a flexible conductive sensitive layer 4 and an electrode array. The electrode array is located at the edge of the flexible conductive sensitive layer, and the flexible conductive sensitive layer and the electrode array are in contact and connected. A monocular vision camera 2 is mounted on the top surface of the gripper base 6.

[0122] The electrode array layout around the flexible conductive sensitive layer can be referenced. Figure 4 Electrodes E1 through E16 are arranged at intervals along the edge of the flexible conductive sensitive layer. When the flexible conductive sensitive layer is compressed, its conductivity changes, which is reflected by the boundary voltage signal collected by the electrode array. The electrode array is specifically numbered E1, E2, E3, E4, E5, E6, E7, E8, E9, E10, E11, E12, E13, E14, E15, and E16; the arrangement of electrodes E1 through E16 at intervals along the edge in the figure indicates that the sixteen electrodes are sequentially arranged at intervals along the edge of the flexible conductive sensitive layer.

[0123] In this embodiment, the EIT flexible tactile sensing unit adopts a multi-electrode array structure, with electrodes spaced apart along the edge of the flexible conductive sensitive layer. The EIT data acquisition module acquires boundary voltage signals according to a preset excitation and measurement method, forming a boundary voltage vector:

[0124]

[0125] in, This represents the boundary voltage vector output by the EIT flexible tactile sensing unit; This represents the m-th boundary voltage measurement value, where m = 1, 2, ..., M; M represents the number of boundary voltage measurements, which can be determined based on the number of electrodes and the excitation measurement method.

[0126] The system acquires the no-load reference voltage vector in a contactless state. During the capture process, the real-time boundary voltage vector is obtained. The voltage change caused by contact is obtained by measuring the difference between the two values.

[0127]

[0128] in, This represents the difference between the real-time boundary voltage vector and the no-load reference voltage vector, i.e., the differential voltage vector caused by contact with the target object. The calculation server obtains the global contact pressure and two-dimensional contact pressure distribution images based on this differential voltage vector.

[0129] robotic arm and end effector Used for visually guided positioning and gripper opening and closing control. The robotic arm moves above the target object based on the target physical position output by the edge actuator, and further moves to the gripping pose; the gripper performs closing, releasing, or holding actions based on the gripper control quantity output by the computing server.

[0130] Communication Link This is used to transmit visual prior information, EIT processing results, gripper control commands, and system status information between the edge execution unit and the computing server. The communication link can be implemented using serial communication, network communication, or other wired / wireless communication methods. The communication link only needs to meet the data interaction requirements during the grasping process and is not limited to a specific communication protocol or hardware interface.

[0131] The data flow relationships of the system can be summarized as follows:

[0132]

[0133] in, This represents the scene image acquired by the visual acquisition module; , This indicates the center coordinates of the target pixel in the image coordinate system; Indicates the material category of the target to be captured; , This represents the physical coordinates of the target object in the robot arm's base coordinate system. Represents the EIT differential voltage vector; Indicates global contact pressure; This represents a two-dimensional contact pressure distribution image. Represents a multidimensional tactile feature vector; Indicates the gripper control quantity; This indicates the degree of opening and closing of the grippers.

[0134] Through the above system architecture, the edge execution end is responsible for target recognition, coordinate mapping and robotic arm motion guidance, the computing server is responsible for EIT tactile data processing, contact pressure image reconstruction and closed-loop force control, the EIT flexible tactile sensing unit provides real-time tactile feedback during the contact process, and the robotic arm and end effector complete the stable grasping of the target object according to the control command.

[0135] II. Visual Prior Acquisition Step S1

[0136] The edge actuator controls the robotic arm to enter a global observation pose and acquires scene images of the target working area through a monocular vision sensor. In this embodiment, the monocular vision sensor is located at the end effector of the robotic arm.

[0137] The acquired scene images are input into a visual recognition model to obtain the recognition result of at least one candidate target. The recognition result of each candidate target includes the target bounding box, material category, and recognition confidence score, which can be represented as:

[0138]

[0139] in, D This represents the set of candidate targets output by the visual recognition model. Indicates the first i One candidate target; Indicates the first i The bounding boxes of the candidate targets; Indicates the first i The material category of each candidate target; Indicates the first i The confidence level of identifying each candidate target; This indicates the number of candidate targets.

[0140] The bounding box It can be represented as:

[0141]

[0142] in, , Let x and y represent the x and y coordinates of the top-left corner of the bounding box of the i-th candidate target, respectively. , They represent the first i The x and y coordinates of the bottom right corner of the bounding box of each candidate target.

[0143] When there are multiple candidate targets in the scene image, the edge execution terminal determines the current target to be captured from the candidate target set D based on the recognition confidence score:

[0144]

[0145] in, This represents the current target to be captured; argmax represents the confidence level required for recognition. The largest candidate target.

[0146] Based on the current target to be captured bounding box Calculate the center coordinates of the target pixel:

[0147]

[0148]

[0149] in, This indicates the horizontal center coordinates of the target to be captured in the image coordinate system; This indicates the vertical center coordinates of the target to be captured in the image coordinate system; , This indicates the coordinates of the top-left corner of the bounding box of the target to be captured. , This indicates the coordinates of the bottom right corner of the bounding box of the target to be captured.

[0150] The visual recognition model also outputs the material category of the target to be grasped. The material category It can include one or more of the following materials: rubber, metal, glass, wood, and plastic, and can also be expanded to other material categories depending on the actual application scenario.

[0151] The edge execution end determines the material type of the target to be captured. Query preset material - retrieve parameter mapping table Determine the target grasping pressure threshold and target contact area threshold corresponding to the current target to be grasped:

[0152]

[0153] in, This indicates the target capture pressure threshold for the current target to be captured; This indicates the target contact area threshold of the target to be captured. This represents the material-fetching parameter mapping table; This indicates the material type of the target to be grasped. For fragile or easily deformable materials, the target grasping pressure threshold is set to a lower value; for rigid materials or materials with high surface friction, the target grasping pressure threshold is set to a higher value.

[0154] In this embodiment, the material-grabbing parameter mapping table Simultaneously output the target contact area threshold. .

[0155] in, This represents the target contact area threshold under a stable grasping state. The target grasping pressure threshold... Contact area threshold with target The gripper is used as the target for subsequent EIT tactile closed-loop control.

[0156] To reduce the impact of short-term occlusion, reflections, or image detection jitter on the grasping process, a short-term visual memory mechanism is implemented at the edge execution end. Let the current time be... The most recent effective target identification time was The set visual memory time window is When no target object is detected in the current image frame, and the following conditions are met:

[0157]

[0158] The edge execution end retains the most recent valid identification result as the prior of the current target; when:

[0159]

[0160] If the edge execution terminal determines that the target is lost, it will stop the current capture task or re-execute the visual recognition.

[0161] It should be noted that the target gripping pressure threshold and target contact area threshold obtained from the material category query in this step are not used as the open-loop pressure command executed by the gripper instantaneously. These thresholds are only used as initial reference values ​​in the subsequent closed-loop force control steps. The actual gripper control quantity is jointly determined by the real-time global contact pressure and multi-dimensional tactile features from the EIT tactile feedback. Therefore, even if there is a certain deviation in visual material recognition, this deviation only affects the initial reference value of control and will not directly cause overpressure or slippage at the moment of gripping.

[0162] Through the above steps, the edge execution unit obtains visual prior information about the target to be grasped. This visual prior information includes at least the target pixel center coordinates, material type, target grasping pressure threshold, and target contact area threshold. This visual prior information is used in subsequent visual guidance and positioning steps and EIT haptic closed-loop control steps.

[0163] III. Visual Guidance Positioning Step S2

[0164] After completing the visual prior acquisition, the edge execution end converts the position of the target object in the image coordinate system into the target physical position in the robot arm base coordinate system based on the target pixel center coordinates of the current target to be grasped, and controls the end gripper of the robot arm to move to the vicinity of the target object.

[0165] This embodiment uses a calibration mapping method to establish the positional correspondence between the image coordinate system and the robot arm base coordinate system. Multiple calibration points are pre-selected on the working plane, and the pixel coordinates of each calibration point in the image coordinate system and its physical coordinates in the robot arm base coordinate system are recorded. The calibration points can be represented as:

[0166]

[0167]

[0168] in, Indicates the first i The pixel coordinates of each calibration point in the image coordinate system; , These represent the x-coordinate and y-coordinate of the pixel at the calibration point, respectively. Indicates the first i The physical coordinates of each calibration point in the robot arm's base coordinate system; , These represent the horizontal and vertical coordinates of the calibration point in the robot arm's base coordinate system, respectively.

[0169] Based on the multiple calibration points, the coordinate transformation matrix from the image coordinate system to the robot arm base coordinate system is obtained. The coordinate transformation relationship can be expressed as:

[0170]

[0171] in, H This represents the coordinate transformation matrix determined by the calibration points; , This indicates the center coordinates of the target pixel of the target to be captured. x '、 y '、 w ' represents the homogeneous coordinate components after coordinate transformation.

[0172] Normalize the homogeneous coordinates to obtain the target's physical coordinates in the robot arm's base coordinate system:

[0173]

[0174]

[0175] in, This indicates the lateral coordinates of the target to be grasped in the robot arm's base coordinate system; This indicates the longitudinal coordinates of the target to be grasped in the robot arm's base coordinate system.

[0176] The edge actuator generates a motion target for the robotic arm's end effector gripper based on the target's physical coordinates. To avoid collisions between the robotic arm and the target object or worktable during approach, the robotic arm first moves to a hovering position above the target object.

[0177]

[0178] in, This indicates the hovering pose of the gripper at the end of the robotic arm; Indicates hovering height; , , This indicates the attitude angle of the gripper at the end of the robotic arm.

[0179] Once the end effector gripper of the robotic arm reaches the hovering pose, the edge actuator controls the robotic arm to move towards the target object to the grasping pose:

[0180]

[0181] in, This indicates the grasping posture of the gripper at the end effector of the robotic arm; Indicates the grab height. Greater than This allows the robotic arm's end gripper to first reach above the target object, and then gradually approach the target object.

[0182] In this embodiment, the guiding process of the robotic arm can be represented as follows:

[0183]

[0184] in, This indicates the current pose of the end effector gripper of the robotic arm; Indicates the hovering pose; This indicates the grasping pose.

[0185] To determine whether the robotic arm has reached the target position, the edge execution end calculates the pose error between the current pose of the robotic arm's end effector and the target pose:

[0186]

[0187] in, Indicates pose error; This indicates the actual pose of the gripper at the end effector of the robotic arm. This represents the target pose corresponding to the current motion phase, where the target pose is... or .

[0188] When the pose error Less than the specified pose error threshold At that time, it is determined that the end effector gripper of the robotic arm has reached the corresponding target position:

[0189]

[0190] in, This indicates the pose error threshold. After the robotic arm reaches the grasping pose, it proceeds to the subsequent EIT tactile sensing and closed-loop force control steps.

[0191] It should be understood that the coordinate transformation matrix determined by the calibration points... H Used to establish the working plane in the image coordinate system and the robot arm base coordinate system. Z The positional correspondence between the =0 plane and the working plane only assumes the lateral coordinates within the working plane. xand vertical coordinates y The coarse localization function is not used for retrieving target objects from monocular images. Z Towards depth. The end effector gripper of the robotic arm is in Z Directional movement is determined by a preset hovering height. and grab height Decision, and in accordance with The phased path execution, in which Greater than This allows the gripper to first move to a hovering position above the target object, and then gradually approach the grasping position to avoid collision problems caused by changes in the thickness of the target object.

[0192] IV. EIT Tactile Perception and Pressure Quantification Steps S3

[0193] Once the robotic arm's end effector gripper moves to the grasping position, the edge actuator drives the gripper to gradually close, bringing the EIT flexible tactile sensing unit, located on the inner wall of the gripper, into contact with the target object. The EIT flexible tactile sensing unit includes a flexible conductive sensitive layer and multiple electrodes disposed at the edge of the flexible conductive sensitive layer. When the gripper contacts the target object, the flexible conductive sensitive layer deforms under pressure, causing a change in its local conductivity distribution, which is reflected by a boundary voltage signal.

[0194] The EIT data acquisition module acquires voltage signals at the boundary of the flexible conductive sensitive layer according to the adjacent excitation and measurement method, and forms a real-time boundary voltage vector:

[0195]

[0196] in, This represents the real-time boundary voltage vector acquired at the current moment. This represents the m-th boundary voltage measurement value, where m = 1, 2, ..., M; This indicates the number of boundary voltage measurements. M can be determined based on the number of electrodes and the excitation measurement method of the EIT flexible tactile sensing unit.

[0197] When the gripper is not in contact with the target object, the system collects the boundary voltage under no-load conditions as a reference voltage. To reduce the influence of random noise, the no-load sampling results can be averaged to obtain the no-load reference voltage vector:

[0198]

[0199] in, Represents the no-load reference voltage vector; Indicates the number of unloaded samples; Let represent the boundary voltage vector obtained from the j-th unloaded sampling.

[0200] During real-time capture, the real-time boundary voltage vector is calculated. With no-load reference voltage vector Differential voltage vector between:

[0201]

[0202] in, This represents the difference between the real-time boundary voltage vector and the no-load reference voltage vector, i.e., the differential voltage vector caused by contact with the target object. This differential voltage vector is used to reduce the impact of sensor initial bias, environmental noise, and temperature drift on the tactile perception results.

[0203] To achieve rapid force feedback, this embodiment calculates the global contact pressure based on the differential voltage vector:

[0204]

[0205] in, Indicates global contact pressure. This represents the pressure quantization function obtained from calibration. In this embodiment, the pressure quantization function can be expressed using the differential voltage norm and the limiting function as follows:

[0206]

[0207] in, Indicates the pressure calibration factor; The L2 norm of the differential voltage vector; This indicates the upper limit of the pressure. The global contact pressure... This is used in subsequent closed-loop force control steps to determine whether the current gripping force of the gripper has reached the target grasping pressure threshold.

[0208] To obtain the spatial pressure distribution at the contact surface, the EIT imaging module uses the differential voltage vector. The conductivity variation distribution within the flexible conductive sensing layer is reconstructed. Under the linearized approximation, the relationship between the differential voltage vector and the conductivity variation distribution can be expressed as:

[0209]

[0210] in, This represents the pre-calculated sensitivity matrix; This represents the distribution of conductivity variation within the flexible conductive sensitive layer. This indicates measurement noise.

[0211] EIT image reconstruction is an inverse problem. This embodiment uses a regularization method to solve for the conductivity variation distribution. Its objective function can be expressed as:

[0212]

[0213] in, Represents the reconstruction objective function; Represents the regularization coefficient; This represents a regularization term used to suppress noise amplification and improve reconstruction stability.

[0214] In this embodiment, the EIT imaging module can use one of the following methods to solve the objective function, namely Tikhonov regularization, Landweber iteration, conjugate gradient least squares algorithm, or linear back projection algorithm, to obtain the conductivity variation distribution. Subsequently, the conductivity variation distribution was mapped to a two-dimensional contact pressure distribution image:

[0215]

[0216] in, This represents a two-dimensional contact pressure distribution image. This represents a pre-calibrated pressure mapping function used to convert the conductivity variation distribution of the flexible conductive sensitive layer into a two-dimensional pressure response image that can be used for tactile feature extraction.

[0217] Through the above steps, the system obtains two types of tactile feedback information: one is global contact pressure. One type is used for rapid closed-loop force control; the other type is two-dimensional contact pressure distribution images. This is used to extract multi-dimensional tactile features such as contact area, peak pressure, and pressure center in subsequent steps.

[0218] V. Multidimensional tactile feature extraction step S4

[0219] After obtaining the two-dimensional contact pressure distribution image, the system performs feature extraction on the image to obtain multi-dimensional tactile features for gripper closed-loop control. The two-dimensional contact pressure distribution image is denoted as... .

[0220] First, the system analyzes the two-dimensional contact pressure distribution image. Threshold segmentation is performed to obtain the effective contact area between the target object and the gripper. The threshold can be a preset threshold or adaptively determined based on the pressure image. The effective contact area can be represented as:

[0221]

[0222] in, Indicates the effective contact area; The horizontal index representing the pixel position in a two-dimensional contact pressure distribution image; The vertical index representing the pixel position in a two-dimensional contact pressure distribution image; Indicates position Pressure response value at the location; This represents the threshold for segmenting the contact area.

[0223] According to the effective contact area Calculate the contact area:

[0224]

[0225] in, Indicates the contact area; This represents the number of pixels in the effective contact area. In this embodiment, the contact area... It can also be converted into the actual physical contact area by setting a calibration coefficient.

[0226] Furthermore, the system extracts the peak pressure from the two-dimensional contact pressure distribution image:

[0227]

[0228] in, This represents the peak pressure within the effective contact area. The peak pressure is used to determine whether there is a risk of localized stress concentration or excessive compression.

[0229] The system also calculates the pressure center based on the pressure distribution within the effective contact area. The pressure center can be represented as:

[0230]

[0231]

[0232] in, and These represent the horizontal and vertical coordinates of the pressure center in the two-dimensional contact pressure distribution image, respectively. The horizontal index representing the pixel position in a two-dimensional contact pressure distribution image; The vertical index representing the pixel position in a two-dimensional contact pressure distribution image; Indicates position The pressure response value at the point; the pressure center is used to characterize the location of the main force-bearing area between the target object and the gripper.

[0233] In this embodiment, the system can determine whether the target object has a slippage tendency based on the changes in the pressure center at adjacent time points. The displacement of the pressure center at adjacent time points can be expressed as:

[0234]

[0235] in, Let represent the pressure center vector at the current moment, and ; This represents the pressure center vector at the previous moment; This represents the Euclidean displacement of the pressure center at adjacent moments. When... If the slippage threshold is exceeded and the global contact pressure has not reached a stable state, the system determines that the target object may have a slippage tendency and provides the judgment result to the subsequent gripper control steps.

[0236] Based on the above characteristics, the system constructs a multi-dimensional tactile feature vector:

[0237]

[0238] in, This represents the multidimensional tactile feature vector at the current moment; Indicates the current contact area; Indicates the current peak pressure; and These represent the horizontal and vertical coordinates of the current pressure center, respectively. This represents the real-time global contact pressure obtained from the EIT pressure quantization step.

[0239] For detailed EIT image reconstruction and multidimensional feature extraction processes, please refer to [link / reference]. Figure 5 As shown, the process includes boundary voltage acquisition, no-load reference voltage acquisition, differential processing, inverse problem solving, pressure mapping, and tactile feature extraction. The system obtains differential voltage and global contact pressure based on real-time boundary voltage and no-load reference voltage, and reconstructs a two-dimensional contact pressure distribution image; then, it extracts contact area, peak pressure, and pressure center to form a multi-dimensional tactile feature column vector.

[0240] Figure 5 This describes the EIT image reconstruction and multidimensional tactile feature extraction process. The upper processing chain sequentially includes boundary voltage acquisition, no-load reference voltage, differential processing, inverse problem solving, pressure mapping, and feature extraction input. The real-time boundary voltage vector is... The no-load reference voltage is obtained in a contactless state, corresponding to... Differential voltage and pressure quantization correspondence and .

[0241] Figure 5 The inverse problem is solved to obtain the conductivity variation distribution, and its linear relationship is: Output Pressure mapping is used to generate a two-dimensional pressure distribution, according to... A two-dimensional contact pressure distribution image is obtained; the feature extraction input simultaneously receives the pressure image and the global pressure, i.e. and .

[0242] Figure 5 Multidimensional tactile feature extraction comprises four branches. The effective contact area is obtained through threshold segmentation. And calculate the contact area. Peak pressure is searched for its maximum value within the effective contact area; the pressure center is calculated using a weighted centroid, where... The horizontal axis is... The vertical axis is given by the pressure center vector. The global contact pressure is quantized by differential voltage, i.e. This is used for rapid closed-loop force control. The above quantities form a multi-dimensional tactile feature column vector: .

[0243] The multidimensional tactile feature vector is used to adjust the gripper opening degree in the subsequent closed-loop force control step.

[0244] VI. Closed-loop force control steps S5

[0245] Once the end effector gripper of the robotic arm reaches the grasping posture, the system enters the closed-loop force control phase. Based on the target grasping pressure threshold determined in the visual prior acquisition step and the real-time tactile feedback information obtained in the EIT tactile perception step, the control module dynamically adjusts the gripper opening and closing degree to ensure that the gripper forms a stable and non-excessive grip on the target object.

[0246] Let the real-time global contact pressure at the current moment be... The target capture pressure threshold is The pressure error at the current moment is:

[0247]

[0248] in, Indicates the current pressure error; This indicates the target gripping pressure threshold determined based on the material of the target object; This represents the real-time global contact pressure obtained from the EIT tactile sensing step.

[0249] The control module based on the pressure error and multidimensional tactile feature vectors Generate gripper opening and closing control values:

[0250]

[0251] in, This indicates the gripper control quantity at the current moment; Represents the closed-loop control function; This represents a multidimensional tactile feature vector at the current moment. The multidimensional tactile feature vector includes at least the contact area, peak pressure, pressure center, and global contact pressure.

[0252] In this embodiment, the closed-loop control function One of the following can be used: PID control, adaptive PID control, fuzzy PID control, impedance control, or model predictive control. Taking PID control as an example, the gripper control quantity can be expressed as:

[0253]

[0254] in, , , These represent the proportional, integral, and derivative control coefficients, respectively. These control coefficients can be preset or adaptively adjusted based on changes in contact area, peak pressure, or pressure center offset.

[0255] In this embodiment, the specific working principle block diagram of the PID controller can be found in the following reference. Figure 6 The PID controller calculates the pressure error based on the target gripping pressure threshold and the real-time global contact pressure, and generates the gripper control quantity through proportional, integral, and derivative components. The gripper opening / closing degree update module drives the gripper actuator to move according to the control quantity, and the EIT tactile perception and feature extraction module feeds back the global contact pressure, contact area, peak pressure, and pressure center to the controller.

[0256] Figure 6 This describes the closed-loop working principle of a PID controller. Target grasping pressure threshold. With pressure feedback Together they are used to calculate the error, resulting in the pressure error. The pressure error is then input into the PID controller.

[0257] A PID controller includes a proportional element, an integral element, a derivative element, and a control output. The proportional element is... The points system is as follows The differential element is The sum of the three is: Multidimensional tactile feedback Used to make at least one control coefficient , , "It can adaptively adjust according to the current tactile characteristics."

[0258] Gripper opening degree update according to Update and meet The gripper actuation module uses this information to perform closing, releasing, or holding actions. The output of the EIT tactile perception and feature extraction module includes... , , , and ,in This forms pressure feedback, and the other features together constitute multidimensional tactile feature feedback.

[0259] Stable crawling criteria include , and Three conditions are considered, and it is determined whether all three conditions are met simultaneously. If not, the closed-loop adjustment continues; if they are met, the current gripper opening degree is maintained and a lifting action is performed.

[0260] The control module updates the gripper opening degree based on the gripper control quantity u(t):

[0261]

[0262] in, This indicates the current gripper opening degree; This indicates the gripper opening / closing degree at the previous moment. It also represents the real-time global contact pressure. Less than the target grasping pressure threshold At that time, the gripper control quantity keeps the gripper closed; when the real-time global contact pressure... Greater than the target grasping pressure threshold At that time, the gripper control amount is adjusted to allow the gripper to relax appropriately.

[0263] To prevent the gripper opening degree from exceeding the allowable range of the mechanical structure, the gripper opening degree must meet the following requirements:

[0264]

[0265] in, Indicates the minimum allowable opening degree of the gripper; This indicates the maximum allowable opening degree of the gripper.

[0266] During closed-loop control, the control module also determines whether the gripping is stable based on multi-dimensional tactile characteristics. When the real-time global contact pressure approaches the target gripping pressure threshold, the contact area reaches the set contact area condition, and the pressure center change is within the allowable range, it is determined that the gripper has formed a stable grip. The stable gripping condition can be expressed as:

[0267]

[0268]

[0269]

[0270] in, Indicates the allowable pressure error; Indicates the current contact area; Indicates the target contact area threshold; Indicates the current pressure center vector; This represents the pressure center vector at the previous moment; This indicates the threshold at which the pressure center can vary.

[0271] like Figure 6 As shown, when the above stable gripping conditions are met, the control module maintains the current gripper opening degree and controls the robotic arm to perform a lifting action.

[0272] If the stable capture conditions are not met within the control time, the system can stop the current closed-loop control or execute a safety handling strategy.

[0273] VII. Communication Protocol and Fault Tolerance / Degradation Mechanism

[0274] During the grasping task execution, the edge execution end and the computing server maintain data interaction. The edge execution end sends visual prior information about the target object to the computing server, and the computing server generates gripper control commands based on EIT haptic feedback and returns the gripper control commands to the edge execution end.

[0275] For details on the communication anomaly degradation and fault tolerance mechanism, please refer to [link / reference]. Figure 7 After completing the visually guided grasping initialization, the system checks the communication status. When communication is normal, the edge execution unit sends material information and the grasping threshold to the computing server, initiates PID control, receives and executes gripper control commands until a stable grasping signal is obtained. When communication is abnormal, the system stops tactile closed-loop force control and enters a safety degradation mode, performing safety actions such as stopping closure, maintaining the current opening degree, releasing the target object, or returning to a safe position. In normal mode, stable grasping and lifting are completed; in degradation mode, safety procedures are completed before the current task ends.

[0276] Figure 7 The process begins with the start of the grasping task and enters the vision-guided grasping initialization phase. Initialization sequentially involves identifying the target object's material, position, and shape; the robotic arm reaching the grasping pose; and reading the target grasping pressure threshold and target contact area threshold. Following this, the system enters the detection and communication state. The state monitoring relationship is as follows: and stipulate .

[0277] When the judgment result indicates that the communication is normal, the system sequentially executes the following steps: sending material information and gripping threshold, starting EIT tactile closed-loop force control, receiving gripper control commands, executing gripper actions, and waiting for a stable gripping signal.

[0278] In normal mode, if the stability determination result is "no", return to execute the gripper action to continue adjustment; if the result is "yes", maintain the opening degree and execute lifting.

[0279] When the judgment result is a communication abnormality, the system sequentially executes the stop tactile closed-loop force control and performs safety procedures according to the current movement stage of the robotic arm: when it is in a safe position, that is, the gripper has not yet performed the lifting action or the target object is within the preset safe release height range, the closing stops, and the target object can be released and the system can retreat to the safe position; when it is not in a safe position, that is, the robotic arm has lifted the target object into the air and the target object is not within the preset safe release height range, the current gripper opening degree and the current position of the robotic arm are maintained, an abnormal alarm is issued and a safety shutdown is executed, without releasing the target object.

[0280] In normal mode, the robotic arm will lift the target object and move it to the target position; in safety downgrade mode, it will end the current task and finally complete the entire grasping process.

[0281] To ensure the security of the data capture process, the system monitors the communication status and EIT data status during the data capture task cycle. The communication status can be represented as:

[0282]

[0283] in, Indicates the system communication and haptic feedback status; This indicates the status of the communication link between the edge execution terminal and the computing server; Indicates the EIT haptic data status; symbol This indicates a logical AND operation. When the communication link is normal... ,otherwise When EIT tactile data is valid ,otherwise .

[0284] When the communication link is normal and the EIT haptic data is valid, the following conditions are met:

[0285]

[0286] The system executes closed-loop force control based on EIT tactile feedback. At this time, the gripper control quantity is generated based on real-time tactile feedback information:

[0287]

[0288] in, Indicates the gripper control quantity; Indicates pressure error; Represents a multidimensional tactile feature vector; This represents the closed-loop control function.

[0289] When the communication link is faulty or the EIT haptic data is invalid, the following conditions must be met:

[0290]

[0291] The system stops the closed-loop force control based on EIT tactile feedback and switches to a preset safety degradation control strategy. The safety degradation control strategy includes one or more of the following: stopping the gripper from closing, maintaining the current gripper opening degree, releasing the target object, performing a preset safety action that limits displacement or force, or controlling the robotic arm to return to a safe position.

[0292] In this embodiment, the gripper opening degree in the degraded mode can be expressed as:

[0293]

[0294] in, This indicates the gripper opening degree in degraded mode; This indicates the preset safety opening / closing degree. It can be set according to the type of target object, gripper structure, or mission safety requirements.

[0295] Through the above communication anomaly handling and degradation control steps, the system can prevent the gripper from continuously applying force, which could damage the target object or cause abnormal robot arm movements, when the communication link is interrupted, the computing server malfunctions, or the EIT tactile data fails, thereby improving the safety of the grasping task.

[0296] 8. Complete Data Capture Task Process

[0297] The complete crawling task flow in this embodiment is as follows:

[0298] Step 1, System Initialization. After system startup, the edge execution terminal, computing server, vision sensing unit, EIT flexible tactile sensing unit, and robotic arm execution mechanism complete initialization. The EIT flexible tactile sensing unit acquires the no-load reference voltage in a non-contact state, obtaining the no-load reference voltage vector. .

[0299] Step 2, Visual Prior Acquisition. The edge execution unit controls the visual sensing unit to acquire images of the target working area, identify the target location and material type of the target to be grasped, and determine the target grasping pressure threshold and target contact area threshold based on the material type:

[0300]

[0301] in, Indicates the target capture pressure threshold; Indicates the target contact area threshold; This represents the material-fetching parameter mapping table; Indicates the material type of the target to be captured.

[0302] Step 3, Visual-Guided Positioning. The edge actuator, based on the target position and coordinate transformation relationship, converts the target object's position in the image coordinate system to its physical position in the robotic arm's base coordinate system, and controls the robotic arm's end effector gripper to move along the following path:

[0303]

[0304] in, This indicates the current pose of the end effector gripper of the robotic arm; Indicates the hovering pose above the target object; This indicates the grasping pose corresponding to the target object.

[0305] Step 4, EIT tactile feedback. After the gripper contacts the target object, the EIT flexible tactile sensing unit acquires the boundary voltage signal in real time and, based on the real-time boundary voltage vector... With no-load reference voltage vector The differential voltage vector is obtained as follows:

[0306]

[0307] The system obtains the global contact pressure based on the differential voltage vector. And reconstruct a two-dimensional contact pressure distribution image. .

[0308] Step 5, Multidimensional Tactile Feature Extraction. The system extracts tactile features from two-dimensional contact pressure distribution images. The contact area, peak pressure, and pressure center are extracted, and combined with the global contact pressure to construct a multi-dimensional tactile feature vector:

[0309]

[0310] in, This represents the multidimensional tactile feature vector at the current moment; Indicates the current contact area; Indicates the current peak pressure; and These represent the horizontal and vertical coordinates of the current pressure center, respectively. This represents the real-time global contact pressure obtained from the EIT pressure quantization step.

[0311] Step 6, Closed-loop force control. The control module generates gripper control parameters based on the target grasping pressure threshold and real-time tactile feedback information:

[0312]

[0313] in, Indicates the gripper control quantity; This represents the pressure error between the target grasping pressure threshold and the real-time global contact pressure. This represents the multidimensional tactile feature vector at the current moment; This represents the closed-loop control function.

[0314] Step 7, Stable Grip Judgment. The system determines whether the gripper has formed a stable grip based on real-time global contact pressure, contact area, and pressure center changes. When the gripping state meets the preset stability conditions, the current gripper opening degree is maintained, and the robotic arm is controlled to perform a lifting action; when the communication status is abnormal or the EIT tactile data fails, the system executes a preset safety degradation control strategy.

[0315] In summary, this embodiment utilizes a collaborative process of visual prior acquisition, visual guided positioning, EIT tactile feedback, multi-dimensional tactile feature extraction, and closed-loop force control to convert the EIT boundary voltage signal into global contact pressure and two-dimensional contact pressure distribution characteristics for robotic arm grasping control. This improves the stability, safety, and adaptability of the robotic arm in grasping target objects of different materials.

Claims

1. A method for intelligent grasping of a robot arm based on fusion of EIT tactile and monocular vision, characterized in that, Includes the following steps: S1: Visual Prior Acquisition: Scene images are acquired using a monocular vision sensing unit to identify the target position and material type of the target object to be grasped, and the target grasping pressure threshold and target contact area threshold are determined based on the material type. The target grasping pressure threshold and target contact area threshold serve as initial reference values ​​for subsequent closed-loop force control steps and are dynamically adjusted based on EIT tactile feedback during actual contact. S2: Visual guidance positioning: Based on the target position, the target physical coordinates of the target object in the base coordinate system of the robotic arm are obtained through coordinate transformation, and the end gripper of the robotic arm is driven to move to the hovering pose and the corresponding grasping pose. The coordinate transformation is used to establish the planar position correspondence between the image coordinate system and the working plane. The movement of the robotic arm along the Z-axis is controlled by the preset hovering height and grasping height. S3: EIT tactile sensing and pressure quantization: During the contact process between the gripper and the target object, the boundary voltage signal of the EIT flexible tactile sensing unit set on the contact surface or inner wall area of ​​the gripper is collected in real time. The global contact pressure is obtained based on the difference between the no-load reference voltage and the real-time boundary voltage, and a two-dimensional contact pressure distribution image is reconstructed. S4: Multidimensional tactile feature extraction: Process the two-dimensional contact pressure distribution image to extract multidimensional tactile features including at least contact area, peak pressure, pressure center and global contact pressure; S5: Closed-loop force control: Generate gripper opening and closing control quantity based on the target gripping pressure threshold, real-time global contact pressure and the multi-dimensional tactile features, and drive the gripper to perform closing, releasing or holding actions until the preset stable gripping conditions are met. S3 includes: The EIT flexible tactile sensing unit includes a flexible conductive sensitive layer and multiple electrodes disposed at the edge of the flexible conductive sensitive layer; The no-load reference voltage vector is acquired in a non-contact state, and the real-time boundary voltage vector is acquired during the grasping process. The differential voltage vector is calculated, and the global contact pressure is obtained based on the differential voltage vector. wherein, represents a pressure quantification function obtained by calibration, represents the difference between the real-time boundary voltage vector and the no-load reference voltage vector, i.e. the differential voltage vector caused by the contact with the target object; The global contact pressure is expressed as: wherein denotes a pressure calibration coefficient; denotes the two-norm of the differential voltage vector; denotes a pressure upper limit value; Based on the differential voltage vector, an inverse EIT problem is constructed, and the conductivity variation distribution is solved using a regularized objective function: wherein, represents a reconstruction objective function; represents a regularization coefficient; represents a regularization term; represents a pre-computed sensitivity matrix; represents a distribution of conductivity change within the flexible conductive sensitive layer; The obtained conductivity variation distribution is mapped to a two-dimensional contact pressure distribution image: wherein represents a two-dimensional contact pressure distribution image; represents a pre-calibrated pressure mapping function; S4 includes: Threshold segmentation is performed on the two-dimensional contact pressure distribution image to obtain the effective contact area: in, Indicates the effective contact area; The horizontal index representing the pixel position in a two-dimensional contact pressure distribution image; The vertical index representing the pixel position in a two-dimensional contact pressure distribution image; Indicates position Pressure response value at the location; Indicates the contact area segmentation threshold; Calculate the contact area and peak pressure based on the effective contact area: wherein, represents the contact area; represents the number of pixel points in the effective contact area, represents the peak pressure within the effective contact area; Calculate the pressure center based on the pressure response distribution within the effective contact area: and x and y represent the lateral and longitudinal coordinates, respectively, of the center of pressure in the two-dimensional contact pressure distribution image; The contact area, peak pressure, pressure center, and global contact pressure are used to construct a multi-dimensional tactile feature vector: wherein, represents a multi-dimensional haptic feature vector at the current time instant; represents the current contact area; represents the current peak pressure; and respectively represent the lateral and longitudinal coordinates of the current pressure center; represents the real-time global contact pressure resulting from the EIT pressure quantification step; S5 includes: The pressure error is calculated based on the target grasping pressure threshold and the real-time global contact pressure. wherein, represents a current pressure error; represents a target grasping pressure threshold determined according to a target object material; represents a real-time global contact pressure resulting from the EIT haptic perception step; Based on the pressure error and the multidimensional tactile feature vector, a gripper opening and closing control value is generated: wherein, represents a gripper control amount at the current time point; represents a closed loop control function; represents a multi-dimensional haptic feature vector at the current time point; The closed-loop control function One of PID control, adaptive PID control, fuzzy PID control, impedance control or model predictive control is adopted. The closed-loop control function With PID control, the control amount of the gripper can be represented as wherein, , , respectively represent proportional, integral and derivative control coefficients, wherein at least one control coefficient is adjustable according to the current multi-dimensional haptic feature vector; Update the gripper opening degree based on the gripper control quantity u(t): wherein, represents the jaw opening degree at the current time point; represents the jaw opening degree at the previous time point; When the following conditions are met simultaneously, the gripper is determined to have formed a stable grip, and the current gripper opening degree is maintained, while the robotic arm is controlled to perform a lifting action: wherein, represents a pressure allowance error; represents a current contact area; represents a target contact area threshold; represents a current pressure center vector; represents a previous pressure center vector; represents a pressure center allowable change threshold.

2. The method of claim 1, wherein the method further comprises: S1 includes: The scene image is input into a visual recognition model to obtain the target bounding box, material category, and recognition confidence of the candidate target. The target to be captured is determined based on the recognition confidence level, and the center coordinates of the target pixels are calculated from its target bounding box. Based on the material category, a preset material-grip parameter mapping table is queried to determine the target gripping pressure threshold and the target contact area threshold; If no target object is identified within the preset time window, the most recent valid identification result is retained as the current target prior. If the target object is not identified within the preset time window, the current grasping task is stopped or the visual recognition is re-executed.

3. The method of claim 1, wherein the method further comprises: S2 includes: Multiple calibration points are pre-selected, and their pixel coordinates in the image coordinate system and their physical coordinates in the robot arm base coordinate system are recorded respectively. The coordinate transformation matrix is ​​then obtained accordingly. The target pixel center coordinates are expanded to homogeneous coordinates, and then normalized after transformation by the coordinate transformation matrix to obtain the target physical coordinates of the target object in the robot arm base coordinate system. The hovering pose and grasping pose of the robotic arm end effector are generated based on the target physical coordinates. When the pose error between the actual pose and the target pose is less than a preset threshold, it is determined that the robotic arm end effector has reached the corresponding target position.

4. The intelligent grasping method for robotic arms based on EIT tactile and monocular vision fusion as described in claim 1, characterized in that, It also includes communication anomaly handling and degradation control steps: During the execution of the capture task, the status of the communication link between the edge execution terminal and the computing server, as well as the status of the EIT haptic data, are monitored. When the communication link is normal and the EIT haptic data is valid, closed-loop force control based on EIT haptic feedback is executed. When the communication link is abnormal or the EIT haptic data is invalid, the closed-loop force control based on EIT haptic feedback is stopped and the preset safety degradation control strategy is switched. The preset safety degradation control strategy includes one or more of the following: stopping the gripper from closing, maintaining the current gripper opening degree, releasing the target object, performing a preset safety action that limits displacement or force, or controlling the robotic arm to return to a safe position.

5. A robotic arm intelligent grasping system based on EIT tactile and monocular vision fusion, implementing the robotic arm intelligent grasping method based on EIT tactile and monocular vision fusion as described in any one of claims 1 to 4, characterized in that, include: The edge execution end is used to perform visual acquisition, target recognition, coordinate mapping and robotic arm motion control, and guide the robotic arm end gripper to the grasping pose corresponding to the target object; The computing server is communicatively connected to the edge execution terminal and is used to perform EIT tactile data acquisition, pressure quantization, contact pressure image reconstruction, multi-dimensional tactile feature extraction, and closed-loop force control. The EIT flexible tactile sensing unit is disposed on the contact surface or inner wall area of ​​the end gripper of the robotic arm, and includes a flexible conductive sensitive layer and multiple electrodes disposed on the edge of the flexible conductive sensitive layer. The robotic arm and end effector are used to perform vision-guided positioning based on the target physical coordinates output by the edge actuator, and to perform gripper actions based on the gripper opening and closing control quantity output by the computing server. The communication link is used to transmit visual prior information, EIT processing results, gripper control commands, and system status information between the edge execution end and the computing server.

6. The intelligent grasping system for robotic arms based on EIT tactile and monocular vision fusion as described in claim 5, characterized in that, The edge execution terminal includes a vision acquisition module, a vision recognition module, a coordinate mapping module, a robotic arm drive module, and a first communication module; The computing server includes an EIT data acquisition module, a pressure quantization module, an EIT imaging module, a tactile feature extraction module, a closed-loop control module, and a second communication module. The edge execution terminal and the computing server constitute a distributed collaborative control architecture.

Images