A multi-modal robotic teaching data acquisition system

Abstract

This invention belongs to the technical field of robot teaching and data acquisition. It discloses a multimodal robot teaching data acquisition system. The teaching device includes an operator input end, an end effector gripper, and a force transmission and feedback mechanism connected sequentially. A tactile sensor is embedded in the force transmission path of the end effector gripper for real-time acquisition of tactile information. A data processing module adjusts the damping parameters of the force transmission and feedback mechanism based on the contact state corresponding to the tactile information acquired by the tactile sensor and the local physical properties of the target object grasped by the end effector gripper. A data generation module performs unified data processing on the operator's input information, the end effector gripper's pose information, the acquired tactile information, and visual data after the damping parameters are adjusted to generate a multimodal robot teaching dataset. This invention achieves impedance adjustment for tactile drive, enabling the system to adapt to various contact states and improving data acquisition quality.

Description

Technical Field

[0001] This invention belongs to the technical field of robot teaching and data acquisition, and more specifically, relates to a multimodal robot teaching data acquisition system. Background Technology

[0002] In the field of robot teaching and data acquisition technology, achieving efficient and reliable human-computer interaction to obtain high-quality training data has always been a core challenge. Traditional robot teaching methods mainly include two categories: direct teaching and teleoperation. Direct teaching involves manually manipulating the robot body to teach its motion trajectory. This method can usually obtain high-precision motion data, but it suffers from limited operating space, slow data acquisition speed, and inability to flexibly handle complex task scenarios. Teleoperation involves the operator indirectly controlling the robot's movement using a control interface or handheld device. It offers flexible operation, but due to the lack of high-precision feedback or multimodal perception information, it is usually difficult to acquire data containing rich environmental interaction features.

[0003] In recent years, the rapid development of deep learning and reinforcement learning has driven data-driven imitation learning strategies, leading to an increasingly urgent demand for high-quality multimodal teaching data. However, existing teaching data acquisition schemes have significant technical shortcomings: poor adaptability. Existing handheld teaching devices (such as UMI) generally use fixed impedance or pure position control methods, lacking a tactile-driven online variable impedance mechanism. When contact off-center loading, slippage precursors, or changes in the stiffness of the target object occur, they cannot actively perform mechanical compensation, resulting in frequent grasping instability or contact interruption during the teaching process. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a multimodal robot teaching data acquisition system, which aims to solve the problem that the poor adaptability of the existing data acquisition system leads to the inability to effectively collect high-quality robot teaching data.

[0005] To achieve the above objectives, according to one aspect of the present invention, a multimodal robot teaching data acquisition system is provided, the system comprising a handheld multimodal teaching device, a data processing module, and a data generation module; The handheld multimodal teaching device includes an operator input terminal, a force transmission and feedback mechanism, and an end effector. The force transmission and feedback mechanism connects the end effector and the operator input terminal. The operator input terminal controls the movement of the end effector through the force transmission and feedback mechanism. The damping parameter of the force transmission and feedback mechanism is adjustable. A tactile sensor is embedded in the force transmission path of the end effector. The tactile sensor is used to collect tactile information from the end effector in real time. The data processing module is used to adjust the damping parameters of the force transmission and feedback mechanism according to the contact state corresponding to the tactile information collected by the tactile sensor and the local physical properties of the target object grasped by the end gripper, so that the force transmission and feedback mechanism can adapt to the current contact state. The data generation module is used to acquire the input information from the operator's input end, the pose information of the end effector, the tactile information of the end effector, and the visual data of the target object after the damping parameters of the force transmission and feedback mechanism are adjusted, and to generate a multimodal robot teaching dataset.

[0006] Furthermore, the end gripper is rotary and is rotatably connected to the force transmission and feedback mechanism; a tactile sensor array formed by multiple tactile sensors is embedded in the interlayer position between the rigid substrate and the flexible finger of the rotary end gripper; the tactile sensors embedded in the interlayer are protected by a flexible protective layer, which is disposed between the tactile sensors and the flexible finger.

[0007] Furthermore, the data processing module is also used to calculate the equivalent interaction force based on the tactile information, and feed the equivalent interaction force back to the operator input terminal, which adjusts the movement of the end effector gripper based on the equivalent interaction force; the formula for calculating the equivalent interaction force is:

[0008] in, For the equivalent stiffness of the force transmission and feedback mechanism, For the equivalent damping of the force transmission and feedback mechanism, , These represent the relative displacement and relative velocity between the operator input end and the end gripper, respectively.

[0009] Furthermore, the data processing module is used to calculate the change in normal contact force at the contact interface formed between the end gripper and the target object. Local indentation of the target object The local equivalent stiffness of the target object is estimated based on the relationship between the tangential disturbance and the normal load at the contact interface, and the stability of the contact interface is estimated based on the relationship between the tangential disturbance and the normal load at the contact interface, thus forming a description of the local physical properties of the target object corresponding to the contact state; the formula corresponding to the local equivalent stiffness is:

[0010] in To prevent constants with a denominator of zero.

[0011] Furthermore, the data processing module is used to perform zero-point calibration, filtering, normalization, and time synchronization processing on the collected tactile information to obtain tactile state quantities. Based on these tactile state quantities, the contact state of the contact process between the end gripper and the target object is identified. The tactile state quantities include the normal contact force. Tangential disturbance, contact area, and pressure center migration Local indentation and slippage risk indicators.

[0012] Furthermore, the data processing module calculates the normal equivalent stiffness of the force transmission and feedback mechanisms according to the following formulas. Normal equivalent damping Tangential equivalent stiffness Tangential equivalent damping Perform the calculation:

[0013]

[0014]

[0015]

[0016] In the formula, , , , The initial impedance parameters of the force transmission and feedback mechanism are the initial normal equivalent stiffness, initial normal equivalent damping, initial tangential equivalent stiffness, and initial tangential equivalent damping, respectively. , This is the adjustment coefficient; The local equivalent stiffness of the target object; To assess the risk of exposure, , The rate of change of contact area. The proportion of edge load, This represents the change in local peak pressure. These are the corresponding weighting coefficients.

[0017] Furthermore, the contact states include a free state, an initial contact state, a stable contact state, an off-center contact state, a slippage precursor state, an overload contact state, and a disengagement contact state. When the number of activated units in the tactile array of the tactile sensor and the contact response are lower than a preset threshold, the contact state is determined to be a free state. When the tactile response first and continuously exceeds the contact threshold, the contact state is determined to be an initial contact state. When the change in the pressure center is less than a preset value and the contact distribution remains stable, the contact state is determined to be a stable contact state. When the load in the edge area increases or the contact area distribution is uneven, the contact state is determined to be an off-center contact state. When the pressure center continuously drifts, the contact area decreases, or the local peak value changes, the contact state is determined to be a slippage precursor state. When the local pressure or the overall contact force exceeds the safety threshold, the contact state is determined to be an overload contact state.

[0018] Furthermore, when When the elevation is increased, the data processing module is used to improve... and At the same time reduce When detected When the stiffness is less than the predetermined value, the data processing module is used to reduce... At the same time, it limits the rate of increase of contact force.

[0019] Furthermore, the data generation module is used to perform unified data processing and quality assessment on the operator's input information, the pose information of the end effector gripper, the collected tactile information, and visual data after the damping parameters of the force transmission and feedback mechanism are adjusted. Based on the assessment results, it filters the data it receives and then generates a multimodal robot teaching dataset based on the filtered data. The quality assessment includes multimodal data time alignment, trajectory smoothness and jump analysis, safety check, multimodal synchronization assessment, and tactile response quality assessment.

[0020] Furthermore, the data generation module is used to assess the tactile response quality for each contact risk event based on synchronously recorded tactile state quantities, contact state labels, impedance parameter changes, and operator input information; wherein, the contact risk event is an identified slippage precursor state, off-center contact state, or overload contact state, and the corresponding contact risk assessment quantity is... Exceeding the preset risk threshold The time window.

[0021] In summary, compared with the prior art, the multimodal robot teaching data acquisition system provided by this invention has the following advantages: 1. The data processing module of this invention adjusts the damping parameters of the force transmission and feedback mechanism based on the contact state corresponding to the tactile information collected by the tactile sensor and the local physical properties of the target object grasped by the end effector, thereby achieving impedance adjustment of tactile drive. This gives the acquisition system better adaptability, enabling stable and effective acquisition of high-quality multimodal robot teaching data. Simultaneously, this invention collects operator input information, end effector pose information, visual data, and force-tactile data during the teaching process, forming a high-quality multimodal robot teaching dataset that can be used for robot strategy learning. This gives the acquisition system better applicability, enabling effective fusion of visual and tactile information in complex operational tasks, improving the usability and learning reliability of the teaching data.

[0022] 2. The tactile sensor is located on the core force transmission path, enabling simultaneous perception of external contact force and internal grasping force. The data processing module further calculates the equivalent interaction force based on the tactile information and feeds this equivalent interaction force back to the operator input. The operator input adjusts the movement of the end effector gripper based on this equivalent interaction force. This allows the operator to perceive the end effector contact state in real time and actively correct their actions, resulting in richer and more stable tactile information, which improves the quality of robot teaching data. Compared to unidirectional open-loop teaching devices, this invention significantly improves the degree of freedom of operation, mechanical perception accuracy, and interactive realism during the teaching process.

[0023] 3. By adjusting the damping parameters of the force transmission and feedback mechanism, the teaching device can provide preset equivalent stiffness or compliance, ensuring operator comfort and safety while achieving high-precision end-effector displacement control, thereby obtaining rich and realistic robot teaching data.

[0024] Compared to traditional teaching schemes that rely solely on position control or fixed impedance, this invention can dynamically adjust the mechanical response characteristics of the teaching device according to the contact state. While suppressing slippage and preventing overload, it ensures operational continuity, significantly improving the safety and adaptability of the teaching process in complex contact tasks. This provides a stable foundation for acquiring robot teaching data and, consequently, improves the quality of robot teaching data.

[0025] 4. The rotating end effector gripper can rotate upon contact force, allowing it to adaptively conform to the surface of the target object to adapt to different operating postures and task scenarios. Specifically, the rotatable end effector gripper constitutes a passive compliance layer of the teaching pendant. Its function is to adapt the spatial geometry in response to instantaneous contact impact—when there is a deviation in posture between the gripper and the target object surface, the gripper passively rotates under the action of contact force to eliminate the geometric mismatch, allowing the contact surface to quickly enter a stable contact state. This passive adaptation process requires no control intervention, has a fast response speed, and can effectively absorb the impact load in the initial contact stage, providing a reliable contact foundation for the stable acquisition of subsequent tactile information.

[0026] 5. The tactile sensor embedded in the interlayer is protected by a flexible protective layer, absorbing impact loads during operation without significantly attenuating the tactile signal, thus improving the durability of the tactile sensor. Simultaneously, a tactile sensor array formed by multiple tactile sensors is embedded in the interlayer between the rigid substrate and the flexible fingers of the end gripper, placing the tactile sensor on the core path of gripper contact force transmission, enabling simultaneous sensing of external contact force and internal grasping force. Specifically, when the flexible fingers are subjected to external contact force, the deformation directly acts on the tactile sensor through the force transmission path, allowing the tactile sensor to simultaneously sense the external contact force between the end gripper and the target object, as well as the internal grasping force generated when the gripper holds the target object. This synchronous acquisition of external contact force and internal grasping force significantly enhances the ability of the teaching data to capture environmental interaction features, improving the completeness and richness of multimodal data.

[0027] 6. When When it rises, it increases. and and reduce To suppress relative slip and attitude instability; when detected When the stiffness is less than the predetermined value, reduce At the same time, the rate of increase in contact force is limited to avoid excessive compression of the target object.

[0028] 7. This invention obtains tactile state quantities by performing zero-point calibration, filtering, normalization, and time synchronization processing on the collected tactile information. Based on these tactile state quantities, contact state recognition is performed on the contact process between the end effector gripper and the target object. The tactile state quantities can be directly used as input for contact state recognition. Compared to traditional methods that only record motion trajectories, this invention significantly improves the expressive power of teaching data in environmental interaction perception, providing rich input information for robot imitation learning and policy learning.

[0029] 8. The contact states of the present invention include free state, initial contact state, stable contact state, off-center contact state, slip precursor state, overload contact state, and disengagement contact state, and provide specific contact state determination conditions. The more detailed classification of contact states and corresponding judgment conditions enable rapid and accurate identification of contact states, and can achieve more precise and accurate adjustment of the damping parameters of the force transmission and feedback mechanism based on the contact state and the local physical properties of the target object grasped by the end gripper.

[0030] 9. The present invention also performs unified data processing and quality assessment on the operator's input information, the pose information of the end effector gripper, the collected tactile information and visual data, and filters the data received by itself based on the assessment results, and then generates a multimodal robot teaching dataset based on the filtered data, which improves the purity of robot teaching data and is beneficial to the generalization ability of downstream policy learning in complex contact tasks. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the rotatable end effector and force transmission and feedback mechanism of the multimodal robot teaching data acquisition system provided in this embodiment of the invention; Figure 2 This is a schematic diagram of the embedded arrangement of the tactile sensor array on the end gripper provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the calculation process for variable impedance provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the unified post-processing and quality assessment process for multimodal teaching data provided in an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0033] This invention provides a multimodal robot teaching data acquisition system. The system breaks through the limitations of traditional unidirectional open-loop teaching. Through the attitude-decoupled end gripper embedded force sensing mechanism and dynamic impedance feedback, it significantly improves the purity and security of teaching data in complex operations, and provides highly robust prior data support for robot strategy learning.

[0034] The system employs a handheld multimodal teaching device integrating a rotatable gripper and an embedded tactile sensor array. Tactile sensors are positioned along the force transmission path at the fingertips of the end effector gripper, enabling simultaneous sensing of external contact force and internal grasping force. A bidirectional dynamic force transmission relationship is established between the operator's input and the end effector gripper, and the closed-loop online adjustment of impedance parameters is driven by real-time acquired tactile information, overcoming the limitations of traditional unidirectional open-loop teaching. Simultaneously, multimodal data collected during the teaching process, including operator input, end effector pose, visual features, and tactile information, undergo unified post-processing and comprehensive quality assessment to generate a high-quality teaching dataset suitable for robot imitation learning or strategy learning.

[0035] The system includes a handheld multimodal teaching device, a data processing module, and a data generation module. The handheld multimodal teaching device includes an operator input terminal, a force transmission and feedback mechanism, and a rotary end effector. The force transmission and feedback mechanism connects the end effector and the operator input terminal. The operator input terminal controls the movement of the end effector through the force transmission and feedback mechanism. The damping parameter of the force transmission and feedback mechanism is adjustable. A tactile sensor is embedded in the force transmission path of the end effector. The tactile sensor is used to collect tactile information from the end effector in real time.

[0036] The data processing module is used to adjust the damping parameters of the force transmission and feedback mechanism based on the contact state corresponding to the tactile information collected by the tactile sensor and the local physical properties of the target object grasped by the end gripper; simultaneously, the data processing module is also used to calculate the equivalent interaction force based on the tactile information and feed the equivalent interaction force back to the operation input terminal; the calculation formula for the equivalent interaction force is:

[0037] in, For the equivalent stiffness of the force transmission and feedback mechanism, For the equivalent damping of the force transmission and feedback mechanism, , These represent the relative displacement and relative velocity between the operator input end and the end gripper, respectively.

[0038] The data processing module is used to identify the contact state based on the tactile information collected by the tactile sensor, estimate the local physical properties of the target object based on the obtained contact state, and adjust the equivalent stiffness and damping parameters of the force transmission and feedback mechanism based on the obtained local physical properties. The data generation module is used to perform unified data processing on the operator's input information, the pose information of the end effector gripper, tactile information, and the collected visual data after the damping parameters of the force transmission and feedback mechanism are adjusted, in order to generate a multimodal robot teaching dataset. Multiple tactile sensors are arranged in an array.

[0039] A bidirectional dynamic force transmission relationship is established between the operator's input end and the end effector gripper to provide high-fidelity tactile feedback and end effector pose. The end effector gripper includes at least one set of rotatable grippers. These rotatable grippers are rotatably connected to a gripper support via a pivot pin, allowing them to passively rotate around the pivot pin when subjected to contact force. This enables the rotatable grippers to adaptively conform to the surface of the target object, adapting to different operating postures and task scenarios. The gripping form of the rotatable grippers can be a two-finger gripper, a three-finger gripper, a dexterous hand, or other types of end effectors.

[0040] A tactile sensor array, consisting of multiple tactile sensors, is embedded in the interlayer between the rigid substrate and the flexible finger of the end gripper. This positions the tactile sensors along the core path of the gripper's contact force transmission, enabling simultaneous sensing of both external contact force and internal grasping force. When the flexible finger is subjected to external contact force, it deforms. This deformation directly acts on the tactile sensor through the force transmission path, allowing the sensor to simultaneously sense both the external contact force between the end gripper and the target object, as well as the internal grasping force generated when the gripper holds the target object. This achieves synchronous acquisition of both external contact force and internal grasping force. The tactile sensor includes at least one sensing element for sensing contact force, pressure distribution, or local deformation. These sensing elements are arranged in an array along the finger's contact surface to cover key contact areas.

[0041] The two-way dynamic mechanical transmission relationship can be realized through mechanical spring damping mechanism, pneumatic mechanism, hydraulic mechanism, electromagnetic mechanism, electric drive active feedback mechanism or a combination of the above methods, providing a hardware foundation for online adjustment of variable impedance based on tactile information.

[0042] The tactile sensor embedded in the interlayer is protected by a flexible protective layer, which is disposed between the tactile sensor and the flexible finger body. The flexible protective layer includes, but is not limited to, a rubber pad, a flexible film, or an elastic material layer, and is used to absorb the impact load during operation without significantly attenuating the tactile sensing signal, thereby improving the durability of the sensor. Preferably, the protective layer may also be provided with waterproof, dustproof, or chemical corrosion resistant functions to adapt to different operating environment requirements.

[0043] The bidirectional dynamic force transmission relationship includes two directions: positive force transmission and negative tactile feedback. The positive force transmission is the operation force input by the operator being transmitted to the end effector via the force transmission and feedback mechanism, driving the end effector to perform clamping, rotation and other operations. The negative tactile feedback is the contact force information collected in real time by the end effector tactile sensor, which is processed and fed back to the operator input in the form of force sensation, so that the operator can directly perceive the establishment of contact, the magnitude of contact force and the changes in contact state between the end effector and the target object.

[0044] By configuring the mechanism parameters, the teaching device can exhibit a preset equivalent stiffness or compliance between the operator's input end and the end gripper. Before or during teaching, the impedance parameters of the force transmission and feedback mechanism are adjusted according to the teaching task type to achieve variable impedance characteristics. For tasks requiring precise force control, a high compliance and low stiffness mode is configured; for tasks requiring rapid response, a high stiffness and low damping mode is configured. The impedance parameter adjustment methods include, but are not limited to, manual setting by the user, automatic task type recognition, or adaptive adjustment based on real-time tactile feedback during teaching. The parameters of the force transmission and feedback mechanism support configuration adjustment, providing a hardware foundation for subsequent online adjustment of variable impedance based on tactile information.

[0045] Before teaching begins, the initial pose of the handheld multimodal teaching device is constrained to achieve spatial alignment between the teaching device coordinate system and the virtual robot base coordinate system. The spatial alignment method includes, but is not limited to, achieving alignment by fixing the initial position of the device, achieving alignment by visual markers or sensor-assisted positioning, or guiding the operator to complete the alignment through a software calibration program.

[0046] When the force on the end effector gripper reaches a preset safety threshold, the response is divided into two levels based on the degree of overload: If the force exceeds the safety threshold but does not reach the dangerous upper limit, it switches to a low-stiffness state; if the force exceeds the dangerous upper limit or continues to rise, it enters a decoupling state. The safety protection trigger has a higher priority than the variable impedance adjustment. Once the force drops below the safety threshold and remains stable, it automatically returns to the normal impedance configuration. The safety threshold is set according to the target object type, task characteristics, and operating environment. The safety protection triggering methods include, but are not limited to, mechanical limit triggering, electronic detection triggering, or force sensor feedback triggering.

[0047] A tactile sensor collects tactile information from the end effector gripper in real time. Based on this information, the contact state is identified. The local physical properties of the target object are estimated based on these contact states. Finally, the equivalent stiffness and damping parameters of the force transmission and feedback mechanism are adjusted based on these local physical properties. These local physical properties include local equivalent stiffness and the coefficient of friction.

[0048] The collected tactile information is subjected to zero-point calibration, filtering, normalization and time synchronization processing, and tactile state quantities for subsequent control and adjustment are constructed based on the preprocessing; the tactile state quantities include, but are not limited to, normal contact force, tangential disturbance, contact area, pressure center migration, local indentation degree, contact interface stability and slip risk index.

[0049] Based on the tactile state quantities, the contact state between the end gripper and the target object is identified. The contact state includes at least the free state, initial contact state, stable contact state, off-center contact state, slip precursor state, overload contact state, and disengagement contact state. At the same time, the local equivalent stiffness of the target object is estimated based on the relationship between the contact force and the local deformation, and the stability of the contact interface is estimated based on the relationship between the tangential disturbance and the normal load, forming a description of the local physical properties of the target object corresponding to the contact state.

[0050] Data processing and quality assessment are performed on the operator's input information, the pose information of the end effector gripper, tactile information, and the collected visual data to generate a high-quality multimodal robot teaching dataset that can be used for robot policy learning.

[0051] The quality assessment includes multimodal data time alignment, trajectory smoothness and jump analysis, security checks, multimodal synchronization assessment, and tactile response quality assessment.

[0052] The present invention will be further described in detail below with reference to specific embodiments.

[0053] Please see Figure 1 This invention provides a tactile-driven closed-loop variable impedance handheld multimodal robot teaching data acquisition system, which includes a handheld multimodal teaching device with an integrated rotatable end gripper. A tactile sensor array is arranged on the force transmission path of the end gripper fingertip to realize the synchronous perception of external contact force and internal grasping force, and to establish a bidirectional dynamic mechanical transmission relationship between the operator input end and the end gripper.

[0054] In a handheld multimodal teaching device, a tactile sensor array is embedded in the force transmission path of the end effector gripper fingertip, and a bidirectional dynamic mechanical transmission relationship is established between the operator input end and the end effector gripper, feeding back the end contact force information to the operator input end.

[0055] Overall structure of the device The handheld multimodal teaching device includes an operator input end, an end gripper, and a force transmission and feedback mechanism connecting the two. The operator input end is configured as an ergonomic handle or operating interface, and the end gripper includes at least one set of rotatable grippers with tactile sensors embedded in them.

[0056] In other embodiments, the overall shape, size, and weight of the device can be adjusted according to the application scenario and task requirements, and the material can be engineering plastics, metal alloys, or composite materials. The gripping form of the end effector includes, but is not limited to, two-finger grippers, three-finger grippers, dexterous hands, or other types of end effectors.

[0057] Rotatable gripper structure The end gripper is a rotatable gripper, which is rotatably connected to the gripper bracket via a pivot pin. When subjected to contact force, it can passively rotate around the pivot pin, so that the gripper contact surface adaptively conforms to the surface of the target object to meet the needs of different operating postures and task scenarios.

[0058] In a preferred embodiment, the rotation range of the rotatable gripper is constrained according to task requirements to prevent operational instability caused by excessive rotation. In other embodiments, the rotation axis direction, rotation range, and reset method of the gripper can be adjusted according to the specific application scenario.

[0059] The rotatable gripper constitutes the passive compliance layer of this device. Its function is to adapt the spatial geometry in response to instantaneous contact impact. When there is a deviation in posture between the gripper and the target object surface, the gripper passively rotates under the action of contact force to eliminate the geometric mismatch, allowing the contact surface to quickly enter a stable contact state. This passive adaptation process requires no control intervention, has a fast response speed, and can effectively absorb the impact load in the initial contact stage, providing a reliable contact basis for the stable acquisition of subsequent tactile information. However, passive compliance can only handle geometric adaptation and cannot dynamically adjust the mechanical response characteristics according to the magnitude, distribution, and trend of the contact force. Therefore, it needs to work in conjunction with the main force transmission and feedback mechanism.

[0060] Tactile sensor array arrangement and protective installation Please see Figure 2 A tactile sensor is embedded in the interlayer between the rigid substrate and the flexible finger of each gripper, placing the tactile sensor on the core path of force transmission. When the flexible finger is subjected to external contact force, it deforms. This deformation acts directly on the tactile sensor through the force transmission path, enabling the sensor to simultaneously sense both the external contact force between the end gripper and the target object, as well as the internal gripping force generated when the gripper holds the target object, thus achieving synchronous acquisition of both external contact force and internal gripping force.

[0061] The tactile sensor includes at least one sensing element for sensing contact force, pressure distribution, or local deformation. The types of sensing elements include, but are not limited to, piezoresistive sensors, capacitive sensors, piezoelectric sensors, optical sensors, or flexible thin-film sensors. The sensing elements are arranged in an array along the contact surface of the finger. The arrangement of the array, the number of sensors, and the density can be designed according to the size, shape, and application requirements of the gripper.

[0062] The tactile sensor embedded in the finger body is protected by a flexible protective layer. This flexible protective layer is disposed between the tactile sensor and the flexible finger body. The material of the protective layer includes, but is not limited to, silicone, polyurethane, rubber, or flexible polymer materials. Its thickness, hardness, and elastic modulus should be optimized according to the sensor characteristics and application scenario to absorb impact loads during operation without significantly attenuating the tactile sensing signal, thereby improving the sensor's durability. In other embodiments, the protective layer may also be equipped with waterproof, dustproof, or chemical corrosion-resistant functions to adapt to different operating environment requirements.

[0063] Establishment of bidirectional dynamic mechanical transmission relationship A force transmission and feedback mechanism is set between the operator input end and the end gripper to establish a two-way dynamic force transmission relationship, which includes two directions: positive force transmission and reverse tactile feedback.

[0064] The positive force transmission is the operation input transmitted to the end gripper through the force transmission and feedback mechanism, forming a force-displacement transmission relationship between the operator's input force and the displacement of the end gripper, driving the end gripper to perform clamping, rotation and other operations.

[0065] Reverse haptic feedback is the contact force information collected in real time by the end-effector haptic sensor, processed and converted into an equivalent interactive force. The equivalent interactive force is fed back to the operator's input terminal in the form of feedback, allowing the operator to directly perceive the establishment of contact, the magnitude of the contact force, and changes in the contact state between the end effector and the target object, and to actively correct the operation accordingly.

[0066] in, For equivalent stiffness, For equivalent damping, , These represent the relative displacement and relative velocity between the operator input end and the end gripper, respectively.

[0067] The force transmission and feedback mechanism can be implemented in ways including but not limited to mechanical spring damping mechanisms, pneumatic mechanisms, hydraulic mechanisms, electromagnetic mechanisms, electric drive active feedback mechanisms, or combinations thereof, with its core adjustable parameter corresponding to the normal equivalent stiffness. Normal equivalent damping Tangential equivalent stiffness and tangential equivalent damping Each parameter has an online configuration interface, supporting real-time writing and updating of values ​​by haptic-driven impedance adjustment. This directly maps the calculated impedance adjustment to the hardware execution layer, forming a complete closed loop from haptic perception to mechanical response. The initial values ​​of the above parameters are set as follows: , , , Before teaching begins, pre-configuration is completed according to the task type: for tasks requiring precise force control, the low initial stiffness mode is configured; for tasks requiring rapid response, the high initial stiffness mode is configured.

[0068] Spatial alignment Before teaching begins, the initial pose of the handheld multimodal teaching device is constrained to achieve spatial alignment between the teaching device coordinate system and the virtual robot base coordinate system.

[0069] Spatial alignment methods include, but are not limited to: achieving alignment through the initial position of a fixed device, achieving alignment through visual markers or sensor-assisted positioning, and guiding the operator to complete alignment through a software calibration program. The alignment accuracy should meet the requirements of subsequent trajectory mapping and strategy learning.

[0070] Security protection mechanism When the force on the end effector gripper reaches a preset safety threshold, a safety protection mechanism intervenes. Its triggering priority is higher than the tactile drive impedance adjustment, and it directly acts on the force transmission and feedback mechanism. This safety threshold is consistent with the overload contact state determination threshold, ensuring consistent linkage between software-level sensing and hardware-level protection.

[0071] Depending on the degree of overload, the safety protection mechanism has two response levels: When the force exceeds the safety threshold but does not reach the danger limit, the force transmission and feedback mechanism switches to a low-stiffness state to flexibly limit further increases in operating force while preserving the operator's basic control over the end effector; when the force exceeds the danger limit or the contact force continues to rise without relief, the force transmission and feedback mechanism enters a decoupled state, completely interrupting the transmission of operating force to the end effector, protecting the operator and equipment safety. The safety protection is triggered by, but is not limited to, mechanical limit triggering, electronic detection triggering, or force sensor feedback triggering.

[0072] After the safety protection is triggered, the system continuously monitors the force level at the end. When the force drops below the safety threshold and remains stable for several sampling cycles, it automatically exits the protection state and returns to the normal impedance configuration, then accepts online adjustment from the force transmission and feedback mechanism again. The recovery process uses a gradual transition of impedance parameters to avoid step fluctuations in operating force caused by sudden switching.

[0073] Real-time acquisition of end-effector tactile information.

[0074] Referring to Figure 3, based on the passive geometric adaptation and stable contact established by the rotatable gripper, the contact process is further dynamically and mechanically adjusted through an active impedance-changing mechanism. During the teaching process, tactile information generated by the contact between the end gripper and the target object is collected in real time. Based on this tactile information, the contact state is identified, the local physical properties of the target object are estimated, and the contact risk is assessed. Furthermore, the impedance parameters of the force transmission and feedback mechanism are adjusted online. , , , This allows for a closed-loop variable impedance regulation process directly driven by tactile information, while simultaneously generating tactile information characterizing the contact process and impedance change process. Compared to the gripper, the force and transmission mechanism operates on a slower timescale, responsible for continuous contact force regulation—increasing damping to suppress instability when off-center loading or slippage precursors appear, and reducing stiffness to prevent overpressure when the object is soft, achieving a two-layer coordinated compliant control of passive geometric adaptation and active dynamic regulation.

[0075] Tactile information acquisition, preprocessing and construction of tactile state quantities During the teaching process, the output signals of the tactile sensors are acquired in real time. The acquired information includes, but is not limited to, the raw response values ​​of each sensing unit, the array activation status, local peak pressure, timestamps, sampling sequence numbers, and the corresponding end effector gripper pose information.

[0076] Preferably, the outputs of multiple tactile sensors are recombined according to their spatial positions to form a contact distribution map characterizing the contact surface of the gripper. Based on this map, information such as contact area, pressure center, pressure peak, edge load distribution, and contact area change trend are extracted to describe the dynamic evolution process between the end gripper and the target object from initial contact, stable contact to contact instability.

[0077] Furthermore, the tactile information undergoes zero-point calibration, filtering, normalization, and time synchronization to suppress noise interference and achieve real-time correspondence with operator input information, end-effector pose information, and visual information. Based on the preprocessing, tactile state quantities for subsequent impedance adjustment are constructed, including but not limited to normal contact force. Tangential disturbance, contact area, and pressure center migration Degree of local indentation The stability of the contact interface and the risk of slippage are indicators. These state quantities will be directly used as inputs for contact state identification and impedance parameter adjustment.

[0078] Contact state identification and local physical property estimation Based on the tactile state parameters, the contact state of the contact process between the end gripper and the target object is identified. The contact state includes at least the following states: free state, initial contact state, stable contact state, off-center contact state, slippage precursor state, overload contact state, and disengagement contact state.

[0079] Specifically, when the number of activated units in the tactile array of the tactile sensor and the contact response are lower than a preset threshold, it is determined to be in a free state; when the tactile response first and continuously exceeds the contact threshold, it is determined to be in an initial contact state; when the pressure center changes little and the contact distribution remains stable, it is determined to be in a stable contact state; when the load in the edge area increases significantly or the contact area distribution is uneven, it is determined to be in an off-center contact state; when the pressure center continues to drift, the contact area decreases, or the local peak changes abnormally, it is determined to be in a slippage precursor state; when the local pressure or the overall contact force exceeds the safety threshold, it is determined to be in an overload contact state, and the safety threshold is consistent with the preset threshold in the safety protection mechanism.

[0080] At the same time, according to the change in normal contact force With local indentation The relationship between the changes in the target object's local equivalent stiffness is used to estimate its local equivalent stiffness.

[0081] in To prevent the use of tiny constants with a denominator of zero, the local equivalent stiffness is described. Reflecting the hardness or softness of the target object, it will be directly used as the normal stiffness. Updated input criteria. Furthermore, the stability of the contact interface is estimated based on the relationship between tangential disturbance and normal load, forming a description of local physical properties corresponding to the contact state.

[0082] Online adjustment and feedback recording of impedance parameters based on tactile information After identifying the contact state and local physical properties, the obtained local equivalent stiffness is comprehensively utilized. The impedance parameters of the force transmission and feedback mechanism are updated online based on the current contact state. A contact risk assessment quantity is constructed based on the tactile state quantities. It is used to comprehensively characterize contact eccentricity, slippage trend, and the degree of local stress anomaly, based on the migration amount from the pressure center. Contact area change rate Edge load ratio and local peak pressure variation Jointly determined:

[0083] in These are the corresponding weighting coefficients. Based on the aforementioned contact risk assessment quantity. Local equivalent stiffness of the target object The four impedance parameters of the force transmission and feedback mechanism are updated online as follows:

[0084]

[0085]

[0086]

[0087] in, , , , These are the initial impedance parameters. , This is the adjustment coefficient. Through the above update method, the local equivalent stiffness of the target object is adjusted. When the contact area is large and the contact state is stable, appropriately increase To enhance the operator's perception of changes in contact; in the event of detection of off-center loading, signs of slippage, or localized abnormal forces leading to... When it rises, it increases. and and reduce To suppress relative slippage and attitude instability; when the target object is detected to be relatively soft... When it is small, decrease It also limits the rate of increase in contact force to avoid excessive compression of the target object.

[0088] Preferably, to prevent frequent fluctuations in impedance parameters due to instantaneous noise, impedance parameter updates are performed only after the contact state has been continuously stable for several sampling periods, and upper and lower limits are applied to the updated impedance parameters, i.e.:

[0089]

[0090] This improves the stability and reliability of the impedance regulation process.

[0091] The updated impedance parameters are written to the online configuration interface of the force transmission and feedback mechanism in real time, enabling the teaching device to output feedback force according to the equivalent impedance relationship. .

[0092] This allows operators to directly perceive contact establishment, local hardness and softness changes, off-center loading trends, and slippage risks during the teaching process, and to proactively correct their actions accordingly. During this process, tactile state quantities, contact state labels, impedance parameter changes, operator input information, end-effector pose information, and corresponding timestamps are recorded simultaneously, forming tactile feature information that includes tactile perception, state recognition, impedance adjustment, and action response relationships, providing input for subsequent multimodal data processing and quality assessment.

[0093] Post-processing and quality assessment of multimodal teaching data: Please refer to Figure 4 The operator input information, end-effector pose information, visual data and tactile information collected during the teaching process are uniformly post-processed and quality evaluated to generate a high-quality multimodal robot teaching dataset.

[0094] Multimodal data time alignment: The collected multimodal data is time-aligned to ensure consistency in the temporal dimension among operator input, end-effector pose, visual data, and tactile information. The time offset of each modality relative to the master time reference is calculated, and this offset can be expressed as absolute, relative, or statistical offset. Time alignment accuracy is evaluated using statistical analysis methods, including but not limited to average offset, standard deviation, maximum offset, and percentile distribution. A time synchronization quality score is calculated for each modality based on the statistical characteristics of the offset. The scoring method can comprehensively consider factors such as the proportion of data within a threshold and the degree of relative offset, using a weighted combination or comprehensive evaluation method to obtain the synchronization quality score. Finally, based on the synchronization quality score, the time alignment quality is divided into multiple levels (e.g., excellent, good, average, poor), providing a basis for subsequent data selection.

[0095] Trajectory smoothness and jump analysis The trajectory smoothness is evaluated and jumps are detected based on the end pose information to identify abnormal situations in the trajectory.

[0096] Kinematic analysis methods are used to evaluate trajectory smoothness. Evaluation methods include, but are not limited to: calculating the higher-order derivative characteristics of the trajectory, analyzing the curvature variation characteristics of the trajectory, and evaluating the frequency domain characteristics of the trajectory. Smoothness evaluation indicators may include the root mean square value of jitter, the rate of change of curvature, and the spectral energy distribution. Based on these indicators, a trajectory smoothness score is calculated, and scoring methods may employ threshold comparison, normalized mapping, or empirical models.

[0097] Jump detection is used to detect abnormal jump phenomena in a trajectory. Methods include, but are not limited to, adjacent frame differential analysis, statistical anomaly detection, and time sequence continuity checks. Detected jump points are marked, and the frequency and severity of jumps are statistically analyzed.

[0098] Trajectory quality comprehensive score: The trajectory quality comprehensive score is calculated by combining the smoothness score and the jump detection results.

[0099] Security check Perform a safety check on the teaching data to ensure it complies with robot kinematic constraints, dynamic constraints, and safe operating procedures. The check includes, but is not limited to, joint limit checks, workspace checks, singular configuration checks, velocity limit checks, acceleration limit checks, torque limit checks, and force perception safety checks. Calculate a safety score based on the safety check results. The scoring method may comprehensively consider factors such as the number of violations and the severity of violations, using a penalty function or deduction mechanism to calculate the score.

[0100] Multimodal synchronous evaluation Assess the synchronization quality between multimodal data to ensure temporal and spatial consistency between different modalities. Assessment items include, but are not limited to: temporal synchronization consistency assessment, force-motion consistency assessment, visual-tactile consistency assessment, and action-motion consistency assessment.

[0101] Tactile response quality assessment During the teaching process, the operator receives the equivalent interaction force between the force and the output of the transmission mechanism. Whether subsequent input actions after feedback constitute an effective risk mitigation response is one of the core indicators for measuring the quality of teaching data. This step specifically introduces a tactile response quality score for the tactile-driven variable impedance mechanism to quantify the operator's actual ability to cope with contact risks in each teaching data point.

[0102] Specifically, based on synchronously recorded tactile state quantities, contact state labels, impedance parameter changes, and operator input information, a response quality assessment is performed on each contact risk event. The contact risk event is defined as an identified slippage precursor state, off-center contact state, or overload contact state, with a corresponding contact risk assessment quantity. Exceeding the preset risk threshold The time window.

[0103] For each risk event window, assess the quality of the operator's response as follows: Preset response time window after a risk event is triggered Internally, the system detects whether the operator's input terminal exhibits one of the following effective corrective actions: active reduction of input amplitude, active adjustment of input pose, or active correction of clamping direction; simultaneously, it detects the contact risk assessment quantity. Does a continuous downward trend occur within the response time window, i.e.:

[0104] in This is the moment a risk event is triggered. If all the above conditions are met simultaneously, the operator's response to the risk event is considered effective and recorded as a successful response; otherwise, it is recorded as an ineffective response.

[0105] Based on this, the tactile response quality score for a single teaching data point is calculated. :

[0106] in The number of risk events successfully resolved The total number of risk events. This is the risk amplitude weighting coefficient. and These are the average risk assessment values ​​after mitigation and at the time of triggering, used to measure the extent of risk mitigation. The tactile response quality score comprehensively reflects the operator's sensitivity to and ability to correct for contact risks such as slippage precursors, off-center loading, and overload during the teaching process.

[0107] Preferably, the tactile response quality score is recorded separately according to risk type, and the slip precursor resolution rate, off-center load correction rate and overload suppression rate are calculated separately to form a fine-grained response quality profile, providing richer prior annotation information for subsequent data screening and strategy learning.

[0108] Comprehensive data quality assessment This includes a comprehensive evaluation of various indicators for individual teaching data, as well as a quality assessment of the dataset as a whole.

[0109] A multi-dimensional evaluation is conducted on each individual teaching data point. Evaluation dimensions include, but are not limited to: data completeness, time-aligned evaluation results, smoothness and jump analysis results, safety check results, synchronization evaluation results, task completion rate, and haptic response quality score. tactile response quality score This includes the total number of risk events, the success rate of resolution, and the quality of classified responses; the comprehensive quality score of a single data point is calculated using methods such as weighted summation, fuzzy comprehensive evaluation, and hierarchical analysis.

[0110] A comprehensive quality assessment of the entire teaching dataset is conducted, including but not limited to: data diversity, data distribution balance, data size, percentage of high-quality data, and task coverage. The overall dataset assessment results can be used to guide subsequent data collection efforts and identify areas with insufficient or weak data.

[0111] High-quality data filtering and dataset generation High-quality data is selected based on the quality assessment results to provide reliable data support for robot policy learning. Selection strategies include, but are not limited to, threshold selection, sorting selection, hierarchical selection, adaptive selection, and outlier removal. For data with minor quality issues but still valuable, data correction or augmentation methods can be used, including but not limited to, trajectory smoothing, time alignment correction, missing value imputation, and outlier correction. The selected high-quality data is organized according to a unified format and structure to generate a standardized multimodal robot teaching dataset. The generated high-quality multimodal robot teaching dataset can be used for robot policy learning, supporting robots in learning fine-grained maneuvering tasks in complex environments and improving the efficiency and performance of policy learning.

[0112] Preferably, the tactile response quality is scored. Teaching data with higher screening weights are prioritized for retention, including teaching data that contains a complete risk triggering and proactive mitigation response process. Dynamic response segments that successfully address slippage warnings, off-center load corrections, or overload suppression have higher demonstrative value for downstream robot strategy learning (such as behavior cloning or diffusion strategies). These data provide the strategy model with correct operational priors in contact risk scenarios, significantly improving the generalization ability and safety of the strategy in complex contact tasks. Teaching data with tactile response quality scores below a preset threshold can be downgraded or removed to avoid introducing teaching segments where operators failed to effectively respond to risks into the training set, thus ensuring the purity of the high-quality teaching dataset.

[0113] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal robot teaching data acquisition system, characterized in that: The system includes a handheld multimodal teaching device, a data processing module, and a data generation module; The handheld multimodal teaching device includes an operator input terminal, a force transmission and feedback mechanism, and an end effector. The force transmission and feedback mechanism connects the end effector and the operator input terminal. The operator input terminal controls the movement of the end effector through the force transmission and feedback mechanism. The damping parameter of the force transmission and feedback mechanism is adjustable. A tactile sensor is embedded in the force transmission path of the end effector. The tactile sensor is used to collect tactile information from the end effector in real time. The data processing module is used to identify the current contact state and the local physical properties of the target object grasped by the end effector based on the tactile information collected by the tactile sensor, and adaptively adjust the damping parameters of the force transmission and feedback mechanism so that the force transmission and feedback mechanism adapts to the current contact state, thereby enabling the handheld multimodal teaching device to operate smoothly; the local physical properties include the local equivalent stiffness of the target object and the stability of the contact interface formed between the end effector and the target object. The data generation module is used to acquire the input information from the operator's input end, the pose information of the end effector, the tactile information of the end effector, and the visual data facing the target object after the damping parameters of the force transmission and feedback mechanism are adjusted, and to generate a multimodal robot teaching dataset.

2. The multimodal robot teaching data acquisition system as described in claim 1, characterized in that: The data processing module is also used to calculate the equivalent interaction force between the end gripper and the target object based on the tactile information, and to feed the equivalent interaction force back to the operator input terminal, which adjusts the action of the end gripper based on the equivalent interaction force. The formula for calculating the equivalent interaction force is: in, For the equivalent stiffness of the force transmission and feedback mechanism, For the equivalent damping of the force transmission and feedback mechanism, , These represent the relative displacement and relative velocity between the operator input end and the end gripper, respectively.

3. The multimodal robot teaching data acquisition system as described in claim 1, characterized in that: The data processing module is used to calculate the change in normal contact force at the contact interface. and the local indentation of the target object The local equivalent stiffness of the target object is estimated by the relationship between the change of the tangential disturbance and the normal load at the contact interface, and the stability of the contact interface is estimated by the relationship between the tangential disturbance and the normal load at the contact interface, so as to obtain the local physical properties of the target object. Local equivalent stiffness The corresponding formula is: in To prevent constants with a denominator of zero.

4. The multimodal robot teaching data acquisition system as described in any one of claims 1-3, characterized in that: The data processing module is used to perform zero-point calibration, filtering, normalization and time synchronization processing on the collected tactile information to obtain tactile state quantities, and to identify the contact state of the contact process between the end gripper and the target object based on the tactile state quantities. The tactile state quantities include normal contact force. Tangential disturbance, contact area, and pressure center migration Local indentation and slippage risk indicators.

5. The multimodal robot teaching data acquisition system as described in claim 4, characterized in that: The data processing module calculates the normal equivalent stiffness of the force transmission and feedback mechanisms according to the following formulas. Normal equivalent damping Tangential equivalent stiffness Tangential equivalent damping Perform the calculation: In the formula, , , , The initial impedance parameters of the force transmission and feedback mechanism are the initial normal equivalent stiffness, initial normal equivalent damping, initial tangential equivalent stiffness, and initial tangential equivalent damping, respectively. , This is the adjustment coefficient; The local equivalent stiffness of the target object; To assess the risk of exposure, , The rate of change of contact area. The proportion of edge load, This represents the change in local peak pressure. These are the corresponding weighting coefficients.

6. The multimodal robot teaching data acquisition system as described in claim 4, characterized in that: The contact states include a free state, an initial contact state, a stable contact state, an off-center contact state, a slippage precursor state, an overload contact state, and a disengagement contact state. When the number of activated units in the tactile array of the tactile sensor and the contact response are below a preset threshold, the contact state is determined to be a free state. When the tactile response first and continuously exceeds the contact threshold, the contact state is determined to be an initial contact state. When the change in the pressure center is less than a preset value and the contact distribution remains stable, the contact state is determined to be a stable contact state. When the load on the edge region increases or the contact area distribution is uneven, the contact state is determined to be an off-center contact state. When the pressure center continuously drifts, the contact area decreases, or the local peak value changes, the contact state is determined to be a slippage precursor state. When the local pressure or overall contact force exceeds a safety threshold, the contact state is determined to be an overload contact state.

7. The multimodal robot teaching data acquisition system as described in claim 5, characterized in that: when When the elevation is increased, the data processing module is used to improve... and At the same time reduce When detected When the stiffness is less than the predetermined value, the data processing module is used to reduce... At the same time, it limits the rate of increase of contact force.

8. The multimodal robot teaching data acquisition system as described in claim 6, characterized in that: The data generation module is used to perform unified data processing and quality assessment on the operator's input information, the pose information of the end effector gripper, the collected tactile information, and visual data after the damping parameters of the force transmission and feedback mechanism are adjusted. Based on the assessment results, it filters the data it receives and then generates a multimodal robot teaching dataset based on the filtered data. The quality assessment includes multimodal data time alignment, trajectory smoothness and jump analysis, safety check, multimodal synchronization assessment, and tactile response quality assessment.

9. The multimodal robot teaching data acquisition system as described in claim 8, characterized in that: The data generation module is used to assess the tactile response quality for each contact risk event based on synchronously recorded tactile state quantities, contact state labels, impedance parameter changes, and operator input information; wherein, the contact risk event is an identified slip precursor state, off-center contact state, or overload contact state, and the corresponding contact risk assessment quantity is... Exceeding the preset risk threshold The time window.

10. The multimodal robot teaching data acquisition system as described in any one of claims 1-3, characterized in that: The end gripper is rotary and is rotatably connected to the force transmission and feedback mechanism; a tactile sensor array formed by multiple tactile sensors is embedded in the interlayer position between the rigid substrate and the flexible finger of the rotary end gripper; the tactile sensors embedded in the interlayer are protected by a flexible protective layer, which is disposed between the tactile sensors and the flexible finger.

Images