Systems and methods for controlling robotic arm grasping with enhanced kinematic imagination

Abstract

This application provides a system and method for enhancing motor imagery in controlling a robotic arm's grasping ability. The method includes: constructing a motor imagery decoding model; obtaining a first mapping relationship; calculating the distance between the mechanical gripper and the target object; classifying the vibration intensity of a vibration unit based on the wearer's perceptible vibration sensation to obtain a vibration grade; establishing a mapping relationship between the distance between the gripper and the target object and the vibration grade to obtain a second mapping relationship; setting the vibration intensity of each unit in the vibration module through the second mapping relationship; encoding vibration feedback instructions based on the first and second mapping relationships, and determining the activated finger vibration area to execute tactile feedback; acquiring the operator's motor imagery adjusted based on tactile feedback, and decoding the grasping intention in real time; if the intention exceeds a set threshold, executing the grasping operation. This application strengthens the association between motor imagery and robotic arm movements through vibration signals, effectively reducing grasping adjustment time.

Description

Technical Field

[0001] This application belongs to the field of brain-computer interface and human-computer interaction technology, and in particular relates to a system and method for controlling a robotic arm to grasp enhanced motor imagination. Background Technology

[0002] Stroke patients with hemiplegia often experience motor dysfunction on the affected side, making it difficult for them to perform daily activities such as grasping and placing objects voluntarily, severely impacting their ability to live independently. Brain-computer interface (BCI) systems, combined with robotic arm assistive technology, offer a new pathway for rehabilitation and functional compensation for these patients. By capturing the patient's motor imagery (MI) signals to control the robotic arm's movement, a closed-loop interaction between the brain, device, and body is established, representing a key technological direction for helping patients rebuild motor perception and improve motor function.

[0003] Precise robotic arm control relies on the user's ability to generate stable and clear EEG signals. Some patients, due to a lack of awareness of the robotic arm's working status or poor spatial imagination, are unable to adjust their interaction strategies in real time, resulting in the robotic arm's inability to accurately grasp objects. Appropriate feedback design can effectively enhance the patient's understanding of the robotic arm's working status.

[0004] Currently, most mainstream robotic arm systems rely on visual feedback for their feedback mechanisms: using cameras or displays to present the spatial position and motion state of the robotic arm to the patient, assisting the patient in adjusting their own motor control intentions to match the robotic arm's movements. While visual feedback, as a mature method of external information transmission, can initially achieve linkage between motor imagination and the robotic arm in structured scenarios, it has significant technical limitations: it can only provide two-dimensional external spatial information and cannot convey intuitive spatial experiences. That is, the tactile feedback required by the patient when controlling the robotic arm (such as imagining the affected hand moving left / right, forward / backward) is difficult to match with the spatial perception logic during the user's internal imagining process.

[0005] Besides the deficiencies in transmitting spatial orientation information, existing systems also lack feedback on distance cues between the robotic arm and the target object. In core tasks such as robotic arm grasping, patients need to adjust the precision of their motor imagery based on the distance between the robotic arm and the target (e.g., precise control of the robotic arm's deceleration at close range, and stable guidance of the movement direction at long range). However, existing visual feedback can only roughly present the distance through images, failing to convert distance information into quantifiable and intuitively perceptible feedback signals. Patients struggle to make precise, distance-dependent MI adjustments through visual judgment, easily leading to deviations in robotic arm grasping movements and further reducing the system's practicality.

[0006] Furthermore, existing BCI robotic arm systems suffer from insufficient feedback and control coordination: most systems have not established a precise mapping mechanism between "robotic arm spatial state - feedback signal - EEG signal," the generation of feedback signals lacks dynamic adaptation to the real-time position of the robotic arm (such as the X / Y / Z axis directions based on the Cartesian coordinate system) and the target distance, and the intensity, position, and other parameters of the feedback signals are not optimized in conjunction with the characteristics of human tactile perception, resulting in low efficiency of feedback information transmission, high learning costs for patients, and difficulty in meeting the needs of long-term rehabilitation and daily use.

[0007] Therefore, there is an urgent need for a new type of feedback technology that can overcome the limitations of visual feedback and solve the technical problems in existing motion-imagination control robotic arm systems, such as the inability of visual feedback to transmit spatial information about the working state of the robotic arm, the lack of distance cues, and the disconnect between imagination and equipment movements, through embodied feedback methods such as tactile vibration. Summary of the Invention

[0008] To address the aforementioned technical problems, this application provides a system and method for controlling the grasping of robotic arms with enhanced motion imagination.

[0009] The technical solution provided in this application is as follows.

[0010] In a first aspect, this application provides a system for controlling a robotic arm to grasp enhanced motion imagination, comprising: An electroencephalogram (EEG) is used to collect a user's brainwave signals, decode motor imagery intentions, and obtain EEG decoding results. An RGB-D camera is used to capture point cloud data of the work scene and calculate the three-dimensional position and distance of the target object to be grasped relative to the end gripper of the robotic arm. The vibration feedback glove includes multiple vibration modules, a control board, and a USB communication interface. The vibration modules are distributed in the finger and palm areas of the glove and are used to provide vibration feedback based on the spatial relationship between the end effector of the robotic arm and the target object. The control board integrates a microcontroller to receive vibration encoding commands from a host computer and drive the vibration modules. Collaborative robotic arms, including grippers, are used to perform movement and grasping actions; The host computer is communicatively linked to the electroencephalogram (EEG) device, the RGB-D camera, the vibration feedback glove, and the collaborative robotic arm; the RGB-D camera is mounted on the gripper. The host computer processes point cloud data from the RGB-D camera, calculates the relative position and distance between the target object and the gripper, generates corresponding vibration-encoded commands, and sends them to the vibration feedback glove to provide real-time tactile cues to the operator. The vibration feedback enables the operator to form a clearer grasping intention during motion visualization, enhancing the perception and decodeability of EEG signals. Based on the EEG decoding results, the host computer determines the operator's grasping timing in real time and controls the closing of the gripper to complete the grasping task. In one possible implementation, the vibration feedback includes finger vibration feedback and palm vibration feedback; wherein, the finger vibration feedback uses at least three fingers to respectively provide feedback on the relative positions of the robotic arm end effector and the target object along the three axes in the Cartesian coordinate system, and the vibration intensity provides feedback on the distance of the relative positions; the palm vibration triggers feedback that the gripper has contacted the target object; the palm vibration and finger vibration simultaneously trigger feedback that the gripping was successful.

[0011] Secondly, this application provides a method for controlling the grasping of a robotic arm with enhanced motion imagination, including: Construct a motion imagery decoding model for recognizing grasping intentions in practical control applications; Establish the mapping relationship between the relative position of the gripper and the target object and the vibration area to obtain the first mapping relationship; Calculate the distance between the mechanical gripper and the target object; The vibration intensity of the vibration unit is graded according to the wearer's perceptible vibration sensation, thus obtaining a vibration grade. Establish a mapping relationship between the distance between the gripper and the target object and the vibration classification to obtain the second mapping relationship; The vibration intensity of each unit in the vibration module is set through the second mapping relationship; Based on the first and second mapping relationships, vibration feedback instructions are encoded, and the activated finger vibration area is determined to perform tactile feedback. It acquires the operator's motor imagination based on tactile feedback and decodes the grasping intention in real time; if the intention exceeds a set threshold, it executes the grasping operation.

[0012] In one possible implementation, the method for establishing the first mapping relationship includes: The vibration regions of the three fingers are mapped to the spatial axes of the Cartesian coordinate system. The vibration regions of the two joints of each finger are mapped to the positive and negative directions of the corresponding spatial axes, respectively. The vibration area of ​​the palm is mapped to the gripper contacting the target object.

[0013] In one possible implementation, calculating the distance between the gripper and the target object includes: Convert the target object within the field of view of the RGB-D camera into point cloud pixels; The point cloud pixels are reduced in data density and denoised, and then plane segmentation is performed. Supporting planes are identified and removed to obtain the first point cloud. The Euclidean cluster extraction algorithm is used to extract the point cloud cluster of the target object from the first point cloud. Based on the fixed transformation matrix from the end effector to the camera and the transformation matrix from the robot base to the end effector, the position of the centroid of the target object in the reference coordinate system is calculated; The Euclidean distance between the gripper and the target object is calculated based on the position of the end effector and the position of the centroid.

[0014] Furthermore, the formula for calculating the position of the centroid of the target object in the reference coordinate system is as follows:

[0015] in, The location of the center of mass; The transformation matrix from the camera to the base. ; This is the transformation matrix from the robot base to the end effector; This is a fixed transformation matrix from the end effector to the camera.

[0016] In one possible implementation, the intensity grading of the vibration intensity of the vibrating unit based on the wearer's perceptible vibration perception includes: The vibration intensity of the vibration unit is graded according to the working voltage range to obtain the initial level; The sensitivity of the ability to distinguish vibration stimuli was obtained through a perception threshold experiment. Clustering is performed based on the sensitivity of the ability to distinguish vibration stimuli, and the initial level is graded to obtain the vibration grade.

[0017] Furthermore, the formula for calculating the sensitivity of the ability to distinguish vibration stimuli is as follows:

[0018] in, The sensitivity to vibrational stimuli is represented by Hits, which is the number of times the stimulus intensity is correctly judged, and FalseAlarm refers to the number of times the stimulus intensity is incorrectly judged. Z represents the inverse function of the standard normal distribution.

[0019] In one possible implementation, clustering the sensitivity to vibration stimuli and classifying the initial levels to obtain vibration classification includes: Collect multiple d' results for each participant at different vibration intensities, and use the average of the d' results as the clustering input; The differences between data points are calculated using Euclidean distance, and a distance matrix is ​​constructed. The Ward method was selected as the clustering strategy, with the goal of minimizing the variance within groups. By merging samples step by step, a dendritic cluster diagram was formed, and the number of clusters was set based on this diagram and the shearing point, thus obtaining the clustering results. The members and characteristics of different clusters in the clustering results were analyzed to determine the vibration intensity boundary points that participants could distinguish, so that each cluster corresponds to a vibration level.

[0020] In one possible implementation, the vibration feedback command is encoded in the following format:

[0021] in, C Encoding vibration feedback commands, Indicates direction encoding. This indicates the vibration classification for the corresponding distance range.

[0022] In one possible implementation, a mapping relationship is established between the distance between the gripper and the target object and the vibration gradation, resulting in a second mapping relationship, including: The distance between the gripper and the target object is divided into distance levels that are the same as the vibration classification levels; By mapping distance levels to vibration classifications, a second mapping relationship is obtained.

[0023] Thirdly, this application provides a device for controlling a robotic arm to grasp enhanced motor imagination, comprising: The building module is used to construct motion image decoding models for recognizing grasping intentions in actual control applications; The first mapping module is used to establish the mapping relationship between the relative position of the gripper and the target object and the vibration area, and to obtain the first mapping relationship. The calculation module is used to calculate the distance between the mechanical gripper and the target object; The vibration grading module is used to grade the vibration intensity of the vibration unit based on the wearer's perceptible vibration sensation, thus obtaining a vibration grade. The second mapping module establishes a mapping relationship between the distance between the gripper and the target object and the vibration level, thus obtaining the second mapping relationship; The setting module is used to set the vibration intensity of each unit in the vibration module through the second mapping relationship; The encoding module is used to encode vibration feedback instructions based on the first and second mapping relationships, and to determine the activated finger vibration area to perform tactile feedback; The execution module is used to acquire the operator's motion imagination based on tactile feedback and decode the grasping intention in real time; if the intention exceeds the set threshold, the grasping operation is executed.

[0024] This application has the following beneficial effects: (1) This application breaks through the limitations of traditional brain-computer interface robotic arm control systems that rely on visual feedback. It constructs additional feedback on the spatial relationship between the robotic arm gripper and the grasping target through a vibration feedback system, thereby enhancing the user's motor imagination ability. By using the precise mapping between the finger joint vibration module and the Cartesian coordinate system, the X / Y / Z axis information of the robotic arm is transformed into embodied tactile signals, matching the somatosensory perception logic in the human motor imagination process, effectively solving the problem of the disconnect between visual feedback and internal motor intention. At the same time, the distance-vibration intensity dynamic matching realized by RGB-D camera point cloud processing provides the user with quantifiable grasping distance cues, significantly enhancing the user's motor imagination ability, especially for users with weaker imagination ability.

[0025] (2) This application possesses strong practicality and clinical translational value, with significant advantages in both technical implementation and user adaptation. The vibration intensity level, determined by human tactile perception threshold calibration, combined with precise PWM control via the STM32 development board, ensures the distinguishability and comfort of the feedback signal. Furthermore, the USB interface communication and ROS topic transmission mechanism achieve low-latency response to vibration feedback, with latency controlled within the user's perception threshold, significantly improving the stability and efficiency of the robotic arm's grasping action. Experimentally, it can be compared to the optimization effect of bidirectional BCI—similar to the characteristic of tactile feedback significantly shortening grasping time. This application strengthens the connection between motor imagery and robotic arm movements through vibration signals, effectively reducing grasping adjustment time, and is particularly suitable for rehabilitation training and daily activity assistance for patients with upper limb dysfunction such as stroke hemiplegia and spinal cord injury. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the vibration feedback system for a robotic arm controlled by motion imagination provided in this application embodiment; in the figure, 101-EEG device, 102-host computer, 103-collaborative robotic arm, 104-RGB-D camera, 105-target object, 106-vibration feedback glove.

[0027] Figure 2 This is a key component in the manufacture of the vibration gloves provided in this application embodiment; in the figure, 201-knuckle vibration module, 202-palm vibration module, 203-control board and 3D printed outer shell; Figure 3 This application provides a mapping relationship between the spatial positional relationship between the robotic arm gripper and the grasped target and the vibration location. Figure 4 This is the vibration intensity classification method provided in the embodiments of this application; Figure 5This application provides a mapping relationship between the distance relationship between the robotic arm gripper and the grasped target and the vibration intensity. Figure 6 This is a schematic diagram of the working process of the vibration glove and robotic arm collaborative system provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the device for controlling the grasping of a robotic arm, which enhances the ability to visualize motion, provided in an embodiment of this application. Detailed Implementation

[0028] The present application will now be described in detail with reference to the accompanying drawings. It should be noted that the described embodiments are for illustrative purposes only and are not intended to limit the scope of the present application.

[0029] As shown in Figure 1, a system for controlling a robotic arm to grasp objects with enhanced kinematic imagination, provided in an embodiment of this application, includes: EEG device 101 is used to collect the user's brain signals, decode motor imagery intentions, and obtain brain decoding results; The RGB-D camera 104 is used to capture point cloud data of the working scene and calculate the three-dimensional position and distance of the target object to be grasped relative to the end gripper of the robotic arm. The vibration feedback glove 106 includes multiple vibration modules, a control board, and a USB communication interface; the vibration modules are distributed in the finger and palm areas of the glove and are used to provide vibration feedback based on the spatial relationship between the end effector of the robotic arm and the target object; the control board integrates a microcontroller to receive vibration encoding instructions from the host computer and drive the vibration modules. The collaborative robotic arm 103 includes grippers for performing movement and grasping actions; The host computer 102 is communicatively linked to the EEG instrument 101, the RGB-D camera 104, the vibration feedback glove 106, and the collaborative robotic arm 103; the RGB-D camera is mounted on the gripper. The host computer processes point cloud data from the RGB-D camera, calculates the relative position and distance between the target object and the gripper, generates corresponding vibration-encoded commands, and sends them to the vibration feedback glove to provide real-time tactile cues to the operator. The vibration feedback enables the operator to form a clearer grasping intention during motion visualization, enhancing the perception and decodeability of EEG signals. Based on the EEG decoding results, the host computer determines the operator's grasping timing in real time and controls the closing of the gripper to complete the grasping task.

[0030] In one possible implementation, the vibration feedback includes finger vibration feedback and palm vibration feedback; wherein, the finger vibration feedback uses at least three fingers to respectively provide feedback on the relative positions of the robotic arm end effector and the target object along the three axes in the Cartesian coordinate system, and the vibration intensity provides feedback on the distance of the relative positions; the palm vibration triggers feedback that the gripper has contacted the target object; the palm vibration and finger vibration simultaneously trigger feedback that the gripping was successful.

[0031] Specifically, the host computer is an industrial computer running Ubuntu 18.04, which communicates with the ROS master node of the Kinova robotic arm via wired Ethernet and connects to the development board via USB interface.

[0032] The host computer's software environment includes: (1) ROS Melodic framework, used for subscribing to robotic arm pose data and issuing vibration commands; (2) OpenCV 4.5 and PCL 1.10 libraries are used for preprocessing and target clustering of RGB-D point clouds; (3) The main control program written in C++ integrates distance calculation, vibration coding and data recording modules to realize real-time task control and data interaction.

[0033] Specifically, the RGB-D camera is an Intel RealSense D435i RGB-D camera or a similar product; Specifically, the collaborative robotic arm is the Kinova Gen 2 lightweight robotic arm or a similar product. A custom 3D-printed bracket is mounted above the end effector of the Kinova Gen 2 robotic arm to ensure that the camera's optical axis is parallel to the direction of movement of the robotic arm's end effector.

[0034] Specifically, the EEG device is a BrainProduct BrainAmp series or similar product.

[0035] Specifically, the communication between the glove and the host computer is achieved through a USB module. The STM32 development board mounted on the glove uses the CDC virtual serial port protocol to send the spatial information of the robotic arm gripper to the host computer connected via USB in real time.

[0036] Figure 2 shows a vibration feedback glove provided in an embodiment of this application. The glove uses an elastic fabric as its base material and employs a modular stitching process to integrate seven micro-vibration modules at corresponding locations on the glove. Specifically, these modules include: The first joint of the thumb (corresponding to the positive direction of the Z-axis) and the second joint of the thumb (corresponding to the negative direction of the Z-axis); The first joint of the index finger (corresponding to the positive direction of the X-axis), and the second joint of the index finger (corresponding to the negative direction of the X-axis); The first joint of the middle finger (corresponding to the positive direction of the Y-axis) and the second joint of the middle finger (corresponding to the negative direction of the Y-axis); Palm area (used to indicate the gripper's touch / grab status).

[0037] Specifically, the micro vibration module is a micro DC vibration motor or a similar product. The motor is encased in a 3D-printed shell, and its connecting wires are encased in insulating material.

[0038] Specifically, the control board of the vibration feedback glove is an STM32 development board or a similar product. As the core of the vibration module control, its circuit design mainly includes: The power management module uses an AMS1117-3.3V LDO to convert the external 5V power supply (USB power or independent power supply) into a stable 3.3V voltage to power the STM32 and vibration module. The PWM drive circuit generates a 1kHz PWM signal through the STM32 TIM1 timer (channels CH1~CH7) to drive the vibration module; The USB communication interface, based on the STM32's built-in USB 2.0 Full-Speed ​​controller, enables low-latency communication with the host computer.

[0039] Figure 3 shows the spatial positional relationship between the robotic arm gripper and the target object and the mapping relationship between the vibration parts provided in the embodiment of this application.

[0040] Based on the spatial orientation of the robotic arm's end effector relative to the target object, the vibration module corresponding to the finger joint is activated, including: (1) Positive X-axis direction: Activate the vibration module of the first joint of the index finger; (2) Negative X-axis direction: Activate the vibration module of the second joint of the index finger; (3) Positive Y-axis direction: Activate the vibration module of the first joint of the middle finger; (4) Negative Y-axis direction: Activate the vibration module of the second joint of the middle finger; (5) Positive Z-axis direction: Activate the vibration module of the first joint of the thumb; (6) Negative Z-axis direction: Activate the vibration module of the second joint of the thumb.

[0041] State-Vibration Module Mapping: Gripper touches target: activates palm vibration module (PWM duty cycle 50%, voltage approximately 3.0V); gripper completes gripping: simultaneously activates all vibration modules (each module PWM duty cycle 80%, voltage approximately 3.5V), continues for 200ms, then resumes normal feedback.

[0042] The following describes a method based on the above system to enhance motion visualization for controlling the grasping of a robotic arm.

[0043] See Figure 4 Methods for enhancing motor imagination to control robotic arm grasping include: S401. Construct a motion imagery decoding model for recognizing grasping and releasing intentions in actual control.

[0044] In one possible implementation, in S401, an electroencephalogram (EEG) is used to collect EEG signals related to motor imagery, and a mapping model between the EEG signals and the grasping / releasing intention is established using a shallow convolutional neural network (Shallow ConvNet).

[0045] Furthermore, S401 includes: S401a. Acquire the EEG signals of motor imagery induced by the operator through actual grasping actions or high-intensity external stimuli (such as videos), and record the EEG data and paired sample data within the grasping duration as training data. S401b: Train the Shallow_Convnet neural network using training data, where the input is real-time EEG signals and the output is the probability of grasping and releasing intentions.

[0046] Specifically, the method for generating EEG signals through motor imagery involves a collection process divided into a preparation phase, a grasping phase, and a placement phase. The main purpose is to collect the temporal variation characteristics of EEG signals under different motor intentions, including: (1) Preparation stage: The operator wears an EEG acquisition device and sits naturally facing the control panel without any physical movement. A target object is placed on the control panel, and the robotic arm is positioned to the side and front, with its end effector gripper performing grasping and placement tasks according to a preset trajectory.

[0047] It should be noted that the EEG and the robotic arm's motion signals remained synchronized throughout the experiment.

[0048] (2) Grabbing phase: During the grasping phase, the collaborative robotic arm gradually approaches the target object as the operator gazes along a pre-set path. The operator expresses the grasping intention simply by naturally focusing on the area between the robotic arm's end effector and the object, and gradually reinforcing the mental image of "closing the gripper and grasping the object." When the robotic arm reaches the pre-grasping position and executes the gripper closure, the operator's mental imagery reaches its peak intensity, corresponding to the maximum stage of the grasping intention signal.

[0049] (3) Placement stage: During the placement phase, the robotic arm moves the object to the target placement point along a preset trajectory. The operator continues to maintain a natural gaze on the area between the robotic arm's end effector and the placement location, gradually reinforcing the visual image of "opening the gripper and placing the object." When the robotic arm reaches the placement point and performs the opening motion, the operator's visual imagery reaches its peak again, completing the entire process of expressing the grasping and placing intention.

[0050] It should be noted that the grasp-and-release intention probability output by the decoding model refers to the probability value corresponding to the grasp and release categories when the model classifies a single EEG sample. The output is [p1, p2], where p1 is the grasp probability and p2 is the release probability. For example, if the probability value output by the model for a given EEG sample is [0.85, 0.15], it means that the model determines the probability of the sample belonging to the grasp category is 85%, and the probability of it belonging to the release category is 15%. The essence of the grasp-and-release intention probability is the model's classification confidence; the higher the probability value for the corresponding category, the higher the model's confidence in classifying the sample into that category.

[0051] S402. Establish the mapping relationship between the relative position between the gripper and the target object and the vibration area to obtain the first mapping relationship.

[0052] In one possible implementation, the method for establishing the first mapping relationship includes: S402a. Map the vibration regions of the three fingers to the spatial axes of the Cartesian coordinate system, where the index finger represents the X-axis, the middle finger the Y-axis, and the thumb the Z-axis. S402b: Map the vibration regions of the two joints of each finger to the positive and negative directions of the corresponding spatial axes, respectively. S402c: Map the vibration area of ​​the palm to the gripper contacting the target object.

[0053] S403. Calculate the distance between the mechanical gripper and the target object.

[0054] It should be noted that the relative position between the gripper and the target object refers to the relative position of the target object's center of mass in space, calculated with the gripper's center point and the tool coordinate system where the robotic arm gripper is located as the reference.

[0055] In one possible implementation, S403 includes: S403a: Converts target objects within the field of view of the RGB-D camera into point cloud pixels; S403b: Reduce the data density and noise of the point cloud pixels, perform plane segmentation, identify and remove the supporting planes to obtain the first point cloud; S403c: The first point cloud is extracted using the Euclidean cluster extraction algorithm to obtain the point cloud cluster of the target object; S403d: Based on the fixed transformation matrix from the end effector to the camera and the transformation matrix from the robot base to the end effector, calculate the centroid position of the target object in the reference coordinate system; S403e: Calculate the Euclidean distance between the gripper and the target object based on the position of the end effector and the position of the centroid.

[0056] Furthermore, the formula for calculating the position of the centroid of the target object in the reference coordinate system is as follows:

[0057] in, The location of the center of mass; The transformation matrix from the camera to the base. ; This is the transformation matrix from the robot base to the end effector; This is a fixed transformation matrix from the end effector to the camera.

[0058] S403 will be described in detail below with reference to specific embodiments.

[0059] See Figure 5 This is a method for calculating the distance between the mechanical gripper and the target being grasped: The distance calculation between the robotic arm's end effector and the target object is achieved through a three-level process: "visual perception - coordinate transformation - distance calculation." This process relies heavily on point cloud data processing from an RGB-D camera and hand-eye calibration results. The specific steps are as follows: (1) Point cloud acquisition and preprocessing: For example, an Intel RealSense D435i RGB-D camera (mounted above the Kinova Gen2 robotic arm end effector in an "eye-on-hand" configuration, with a horizontal field of view of 85°, a vertical field of view of 5°, and a diagonal field of view of 90°) was used to acquire point cloud data of the robotic arm's operating space at a frame rate of 30fps. Subsequently, voxel mesh downsampling was performed, with the voxel size set to 5mm×5mm×5mm. While preserving the geometric structure of the target object, the point cloud data density was reduced to decrease the computational load. The number of neighboring points was set to 50, and the standard deviation multiplier was set to 1.0 to remove isolated noise points in the point cloud and statistical outlier removal. The RANSAC algorithm (maximum number of iterations 1000, distance threshold 0.02m) was used to identify and remove supporting planes (such as a desktop), retaining only non-planar point clouds as candidates for grasping targets.

[0060] (2) Target object clustering and centroid calculation: Euclidean clustering is performed on the preprocessed non-planar point cloud to extract the target object and calculate its centroid. First, based on the KD tree search structure, spatially adjacent point clouds are divided into multiple clusters with a spatial neighborhood radius of 0.05m; then, the cluster with the most points is selected as the target object; according to the formula... Calculate the centroid of the target object in the camera coordinate system, where Let N be the camera coordinates of the i-th point in the cluster, and N be the total number of points in the cluster. As the robotic arm approaches the target, the camera view is updated, new point clouds are collected in real time, and the above steps are repeated to dynamically optimize the centroid accuracy.

[0061] (3) Coordinate transformation and Euclidean distance calculation: Coordinate transformation is achieved through hand-eye calibration results, and the Euclidean distance between the end effector of the robotic arm and the target is finally calculated. A fixed transformation matrix from the end effector of the robotic arm to the camera is obtained in advance through calibration experiments. Simultaneously, the transformation matrix from the robotic arm base to the end effector can be obtained in real time using the Kinova ROS SDK. According to the formula Calculate the transformation matrix from the camera coordinate system to the robot arm base coordinate system. .

[0062] (4) Target centroid in camera coordinate system Through formula Transform to the base coordinate system to obtain The Euclidean distance was calculated using the Kinova ROS SDK to obtain the position of the robotic arm's end effector in the base coordinate system. According to the formula Calculate the Euclidean distance d between the two, which is the actual distance between the robotic arm and the target object.

[0063] S404. The vibration intensity of the vibration unit is graded according to the wearer's perceptible vibration sensation to obtain the vibration grade.

[0064] In one possible implementation, S404 includes: S404a. The vibration intensity of the vibration unit is graded according to the working voltage range to obtain the initial grade; S404b. The sensitivity of the ability to distinguish vibration stimuli is obtained through a perception threshold experiment. S404c: Cluster the sensitivity of vibration stimuli differentiation ability, and classify the initial level to obtain vibration classification.

[0065] Furthermore, the formula for calculating the sensitivity of the ability to distinguish vibration stimuli is as follows:

[0066] in, The sensitivity to vibrational stimuli is represented by Hits, which is the number of times the stimulus intensity is correctly judged, and FalseAlarm refers to the number of times the stimulus intensity is incorrectly judged. Z represents the inverse function of the standard normal distribution.

[0067] Furthermore, S404c includes: (1) Collect multiple d' results for each participant at different vibration intensities, and take the average of the d' results as the clustering input; (2) The difference between each data point is calculated using Euclidean distance, and a distance matrix is ​​constructed; (3) The Ward method is selected as the clustering strategy. The optimization goal is to minimize the variance within the group. By merging samples step by step, a tree-like clustering diagram is formed. Based on this, the number of clusters is set according to the shearing point to obtain the clustering results. The members and characteristics of different clusters in the clustering results are analyzed to determine the vibration intensity boundary points that the participants can distinguish, so that each cluster corresponds to a vibration level.

[0068] S404 will be described in detail below with reference to specific embodiments.

[0069] See Figure 6 Methods for classifying vibration intensity include: The intensity levels of the vibration module are determined through quantitative experiments to identify vibration intensity ranges that can be clearly distinguished by the user. This ensures that the mapping relationship between the intensity level and the distance between the robotic arm and the target conforms to the characteristics of human tactile perception. The specific implementation process is as follows: (1) Hardware voltage gradient setting: Based on the PWM driving capability of the STM32 development board (control board) and the electrical characteristics of the vibration module (minimum activation voltage 2.3V, working voltage 2.5V~4.0V), first preset the voltage gradient and corresponding PWM parameters.

[0070] For example, the subjects first wear gloves, and the voltage is increased in increments of 0.1V from the minimum activation voltage until the minimum sensing threshold is reached. The minimum sensing voltage is then recorded. The average of the minimum sensing thresholds for each subject is calculated to obtain the voltage corresponding to the lowest vibration sensing intensity of 3.3V. The voltage range from the minimum voltage to the maximum operating voltage of the vibration module is then divided equally, and the voltage range of 3.3V to 3.93V is selected as the core test voltage range (avoiding the nonlinear response region of the low voltage segment of the vibration module to ensure that the vibration intensity and voltage are linearly correlated). The voltage is divided into 10 voltage levels with an equal difference of 0.07V. The voltage values ​​of each level are 3.30V, 3.37V, 3.44V, 3.51V, 3.58V, 3.65V, 3.72V, 3.79V, 3.86V, and 3.93V, respectively, and numbered 0 to 9.

[0071] The PWM signal is generated using the TIM1 timer of the STM32. With a system clock of 72MHz, the frequency division factor is set to 71 and the auto-reload value is set to 999. Based on the voltage-duty cycle calibration curve (obtained in advance by acquiring the voltage across the vibration module at different duty cycles through the ADC), the PWM duty cycle corresponding to each voltage level is determined. For example, 3.30V corresponds to a duty cycle of 30% and 3.93V corresponds to a duty cycle of 85%, ensuring that the voltage output error is ≤±0.02V.

[0072] (2) Calibration of human subjective perception: For example, 20 healthy adults (with an age difference of no more than 5 years, tactile sensory impairment, and an equal number of males and females) were selected as subjects. Pre-training was performed to ensure consistency in sensory abilities. Then, the subjects participated in training tasks using a "paired vibration intensity comparison" mode. In each training session, two vibration signals of different voltage levels (denoted as vibration A and vibration B) were randomly selected. Each vibration lasted for 2000 ms, and the interval between the two vibrations was 500 ms. After the vibration ended, the subjects needed to judge the magnitude relationship between vibration A and vibration B. A total of 30 trials were performed, and the subjects' judgment accuracy was calculated. Only subjects with an accuracy of ≥90% were retained to enter the formal experiment. In the formal experiment, the glove was connected to the host computer via USB. A C++ program was used to play a pseudo-random sequence of vibration signals at 10 voltage levels, with each level played 3 times, for a total of 30 formal trials, to avoid the subject's memory of the sequence affecting their judgment. Each vibration signal lasted for 2000ms, and after the playback ended, a 3-second judgment window was triggered. The subject pressed the corresponding keyboard key (0~9 keys) based on their subjective perception, and the judgment result was recorded. During the experiment, only the vibration module of the first joint of the index finger of the vibrating glove was activated (to eliminate interference from multiple modules), and each activation was spaced 2 seconds apart to avoid tactile fatigue.

[0073] Specifically, the program calls the serial.Serial() function to transmit vibration control commands, and at the same time uses the write_to_csv() function to save the "actual voltage level - subject's judgment level - response time" data for each trial in real time. Trials with a reaction time of more than 2 seconds are considered invalid.

[0074] (3) Valid level screening: The subjects' judgment of the trial first needs to be converted into the sensitivity of intensity change. The calculation formula is as follows:

[0075] Z represents the inverse function of the standard normal distribution, and Hits refers to the number of times the subject correctly judges the "signal stimulus" as "vibrating signal". Taking the lowest level V1 as an example, if 27 out of 30 lowest signal stimuli are correctly judged as "vibrating", then the number of Hits at the lowest level is 27, and the Hits probability:

[0076] False Alarm refers to the number of times the subject wrongly judges the "noise stimulus" as "vibrating signal". If 3 out of 30 noise stimuli are wrongly judged as "vibrating", then the number of False Alarm is 3, and the False Alarm probability:

[0077] It is shared by all intensity levels, and all need to perform a hierarchical clustering algorithm for level division, and the voltage ranges that are consistent in clustering are regarded as having no difference in the subjective vibration intensity felt by the subject.

[0078] Similarly, calculate the relevant parameters of V2 and V3.

[0079] S405. Establish a mapping relationship between the distance between the gripper and the target object and the vibration classification, and obtain the second mapping relationship.

[0080] In a possible implementation manner, the S405 includes: S405a. Divide the distance between the gripper and the target object into the same number of distance levels as the number of vibration classification levels; S405b. Map the distance levels to the vibration classification to obtain the second mapping relationship.

[0081] Exemplarily, such as Figure 7 , the vibration intensity of the vibration module in this application is divided into 3 levels, namely V1, V2, and V3, which represent the evenly divided intervals of the maximum grasping range of the robotic arm divided into three equal parts, or the robotic arm grasping distance is divided proportionally according to the comfortable area and the maximum grasping interval of the human sitting posture in the vertical and horizontal grasping spaces.

[0082] Exemplarily, divide the grasping stage according to the distance d and match the vibration intensity: (1) Long-distance stage (300mm < d ≤ 600mm): Vibration intensity level V1, corresponding to a PWM duty cycle of 20% (voltage about 2.5V); (2) Medium-distance stage (100mm < d ≤ 300mm): Vibration intensity level V2, corresponding to a PWM duty cycle of 50% (voltage about 3.0V); (3) Close range stage (0mm ≤ d ≤ 100mm): Vibration intensity level V3, corresponding to PWM duty cycle 80% (voltage approximately 3.5V).

[0083] S406. Set the vibration intensity of each unit in the vibration module through the second mapping relationship.

[0084] S407. Based on the first and second mapping relationships, perform vibration feedback instruction encoding and determine the activated finger vibration area to perform tactile feedback.

[0085] In one possible implementation, the vibration feedback command is encoded in the following format:

[0086] in, C Encoding vibration feedback commands, Indicates direction encoding. This indicates the vibration classification for the corresponding distance range.

[0087] S408. Obtain the operator's motion imagery adjusted based on tactile feedback and decode the grasping intention in real time; if the intention exceeds the set threshold, execute the grasping operation.

[0088] The workflow of the aforementioned method is described below. See [link / reference] Figure 6 The workflow includes: Step 1: System Initialization and Calibration. After the host computer starts, hand-eye calibration is automatically triggered. Ten sets of pose data between the robotic arm's end effector and the RGB-D camera are collected using the AprilTag calibration board. The transformation matrix from the camera to the robotic arm base is then calculated. .

[0089] Step 2: Acquisition and Decoding of Motor Imagery Signals. The BCI device acquires the user's motor imagery (MI) EEG signals, filters and extracts features, and then inputs them into the trained recognition model. The signals are decoded to obtain the basic movement commands of the robotic arm (such as movement along the X-axis and gripper closure), and the commands are transmitted to the control center module of the host computer.

[0090] Step 3: Robotic arm and target state perception. The Intel RealSense D435i camera acquires point cloud data of the operational space at a frame rate of 30fps. The host computer calls the PCL library to perform voxel downsampling, plane segmentation and Euclidean clustering to extract the centroid of the target object and transform it to the coordinate system of the robotic arm base. At the same time, the pose of the end effector is obtained in real time through the / j2n6s300 / end_effector_pose topic, and the Euclidean distance d between the two is calculated.

[0091] Step 4: Vibration Feedback Command Encoding. The host computer matches the vibration intensity based on the distance d (d>300mm corresponds to V1, 100mm≤d≤300mm corresponds to V2, d<100mm corresponds to V3), and determines the activated finger joint module based on the spatial orientation of the robotic arm relative to the target (e.g., activating the first joint of the index finger in the positive X-axis direction). The command is encoded in the format of "direction code + intensity level + status code" (e.g., positive X-axis direction + V2 corresponds to 0x01 0x02 0x00).

[0092] Step 5: Command Transmission and Tactile Feedback Execution. The encoded command is sent to the STM32 development board via the CDC virtual serial port protocol. After parsing the command, the development board controls the LEDC peripheral to output a PWM signal with the corresponding duty cycle, driving the designated vibration module of the vibrating glove to work, converting the spatial state of the robotic arm into a embodied tactile signal.

[0093] Step 6: Robotic Arm Action Execution and Status Feedback. The operator corrects the EEG control signal through vibration feedback, generates robotic arm motion control commands, and sends them to the Kinova Gen2 robotic arm via the ROS driver package to execute movement, approach, and grasping actions. When the robotic arm gripper touches the target, it triggers the activation of the palm vibration module (status code 0x01) to indicate the contact status. After the grasping is completed, it triggers full module vibration to indicate the grasping intention, forming a closed-loop feedback.

[0094] The following describes a test method to determine whether the aforementioned method enhances the wearer's motor imagination.

[0095] Due to individual differences among wearers, the enhancement effect varies from person to person, so this testing method can evaluate the enhancement effect.

[0096] The intention decoding test for motor imagery was achieved using the FBCSP+SVM model and ERD feature analysis of EEG signals. The FBCSP+SVM model was used to test the impact of vibration feedback on the EEG discriminative power of motor imagery. The FBCSP model first decomposed the raw EEG signal using multi-band filtering to extract μ and β rhythm features related to grasping and relaxation motor imagery. After dimensionality reduction enhancement using co-space patterns, a high-discriminative feature set was obtained. Then, high-dimensional linear classification was achieved through SVM kernel function mapping, completing the decoding of grasping intentions.

[0097] Event-related desynchronization (ERD) is a physiological phenomenon characterized by a decrease in the electroencephalographic power of the μ and β rhythms in the sensorimotor area during cortical activities such as motor imagery. It is also a core physiological indicator for extracting motor intention-related EEG features from motor imagery brain-computer interfaces. A stronger ERD phenomenon is observed at the C3 electrode location in the contralateral brain region when performing or imagining grasping actions. ERD features can be used to analyze the impact of this glove feedback method on intention stability and signal quality.

[0098] For example, glove wearers were asked to complete a motor imagery task under both haptic and non-haptic feedback conditions. During this process, EEG data was collected and processed through a 1-30Hz bandpass filter and a 50Hz power line interference removal filter. The data then underwent ICA artifact removal, downsampling, bad segment removal, baseline correction, and segmentation before being exported and fed into the FBCSP model for feature extraction. In FBCSP, the filter bank was set to 7-12Hz, 12-16Hz, 16-20Hz, 20-24Hz, 24-28Hz, and 28-30Hz for filtering. Relevant features were calculated from the filtered data. Feature selection could utilize mutual information, approximate entropy, etc. For example, the formula for information entropy is:

[0099] Joint probability distribution and This represents a marginal probability distribution. When X and Y are completely independent, mutual information... The stronger the dependence between the two, the greater the mutual information value. The feature vectors concatenated based on this mutual information are then classified by a Support Vector Machine (SVM) classifier. SVM is a supervised learning algorithm whose core objective is to find the optimal hyperplane in the training set to effectively separate positive and negative samples while maximizing the margin between this hyperplane and the nearest neighbor samples of each class. Its calculation formula is:

[0100] : Optimal Lagrange multipliers; : Labels of the training samples; Kernel function; : Bias term; : The sign function, which outputs +1 or -1 to indicate the category to which the sample belongs.

[0101] If the accuracy of EEG data classification is higher under vibration feedback, it indicates that the glove feedback can help distinguish between EEG in grasping and relaxed states, thereby improving the accuracy of intent recognition.

[0102] It should be noted that the decomposition function includes, but is not limited to, SVM, and can also use models with architectures such as ANN and CNN.

[0103] In ERD analysis, if the time-frequency EEG pattern accompanied by vibration feedback shows a stronger ERD phenomenon in the 8-13 Hz frequency band and lasts longer, it indicates that this feedback method helps the wearer maintain a clearer and more sustained motor intention.

[0104] The apparatus for controlling a robotic arm to grasp enhanced kinematic imagination provided in this application will be described below. The apparatus for controlling a robotic arm to grasp enhanced kinematic imagination described below can be referred to in correspondence with the method for controlling a robotic arm to grasp enhanced kinematic imagination described above.

[0105] Figure 7 This is a schematic diagram of the structure of the device for controlling the grasping of a robotic arm, as provided in the embodiments of this application, to enhance the ability of motion visualization. Figure 7 As shown, it includes: a construction module 71, a first mapping module 72, a calculation module 73, a vibration grading module 74, a second mapping module 75, a setting module 76, an encoding module 77, and an execution module 78, wherein: Module 71 is used to build a motion image decoding model for recognizing grasping intentions in actual control. The first mapping module 72 is used to establish the mapping relationship between the relative position between the gripper and the target object and the vibration area, and to obtain the first mapping relationship. Calculation module 73 is used to calculate the distance between the mechanical gripper and the target object; The vibration grading module 74 is used to grade the vibration intensity of the vibration unit according to the wearer's perceptible vibration sensation, and obtain the vibration grading. The second mapping module 75 establishes a mapping relationship between the distance between the gripper and the target object and the vibration classification, thus obtaining the second mapping relationship; Setting module 76 is used to set the vibration intensity of each unit in the vibration module through the second mapping relationship; The encoding module 77 is used to encode vibration feedback instructions based on the first and second mapping relationships, and to determine the activated finger vibration area to perform tactile feedback; The execution module 78 is used to obtain the operator's motion imagination adjusted based on tactile feedback and decode the grasping intention in real time; if the intention exceeds the set threshold, the grasping operation is executed.

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A system for controlling a robotic arm to grasp enhanced kinematic imagination, characterized in that, include: An electroencephalogram (EEG) is used to collect a user's brainwave signals, decode motor imagery intentions, and obtain EEG decoding results. An RGB-D camera is used to capture point cloud data of the work scene and calculate the three-dimensional position and distance of the target object to be grasped relative to the end gripper of the robotic arm. The vibration feedback glove includes multiple vibration modules, a control board, and a USB communication interface. The vibration modules are distributed in the finger and palm areas of the glove and are used to provide vibration feedback based on the spatial relationship between the end effector of the robotic arm and the target object. The control board integrates a microcontroller to receive vibration encoding commands from a host computer and drive the vibration modules. Collaborative robotic arms, including grippers, are used to perform movement and grasping actions; The host computer is communicatively linked to the electroencephalogram (EEG) device, the RGB-D camera, the vibration feedback glove, and the collaborative robotic arm; the RGB-D camera is mounted on the gripper. The host computer processes point cloud data from the RGB-D camera, calculates the relative position and distance between the target object and the gripper, generates corresponding vibration-encoded commands, and sends them to the vibration feedback glove to provide real-time tactile cues to the operator. The vibration feedback enables the operator to form a clearer grasping intention during motion visualization, enhancing the perception and decodeability of EEG signals. Based on the EEG decoding results, the host computer determines the operator's grasping timing in real time and controls the closing of the gripper to complete the grasping task.

2. The system according to claim 1, characterized in that, The vibration feedback includes finger vibration feedback and palm vibration feedback; wherein, the finger vibration feedback uses at least three fingers to provide feedback on the relative position of the robotic arm end effector and the target object on the three axes of the Cartesian coordinate system, and the vibration intensity provides feedback on the distance of the relative position; the palm vibration triggers feedback that the gripper has contacted the target object; the simultaneous activation of palm vibration and finger vibration provides feedback that the gripping was successful.

3. A method for enhancing motor imagination to control a robotic arm's grasping ability, characterized in that, The method is implemented based on the system described in claim 1 or 2, and the method includes: Construct a motion imagery decoding model for recognizing grasping intentions in practical control applications; Establish the mapping relationship between the relative position of the gripper and the target object and the vibration area to obtain the first mapping relationship; Calculate the distance between the mechanical gripper and the target object; The vibration intensity of the vibration unit is graded according to the wearer's perceptible vibration sensation, thus obtaining a vibration grade. Establish a mapping relationship between the distance between the gripper and the target object and the vibration classification to obtain the second mapping relationship; The vibration intensity of each unit in the vibration module is set through the second mapping relationship; Based on the first and second mapping relationships, vibration feedback instructions are encoded, and the activated finger vibration area is determined to perform tactile feedback. It acquires the operator's motor imagination based on tactile feedback and decodes the grasping intention in real time; if the intention exceeds a set threshold, it executes the grasping operation.

4. The method according to claim 3, characterized in that, The method for establishing the first mapping relationship includes: The vibration regions of the three fingers are mapped to the spatial axes of the Cartesian coordinate system. The vibration regions of the two joints of each finger are mapped to the positive and negative directions of the corresponding spatial axes, respectively. The vibration area of ​​the palm is mapped to the gripper contacting the target object.

5. The method according to claim 3, characterized in that, The calculation of the distance between the gripper and the target object includes: Convert the target object within the field of view of the RGB-D camera into point cloud pixels; The point cloud pixels are reduced in data density and denoised, and then plane segmentation is performed. Supporting planes are identified and removed to obtain the first point cloud. The Euclidean cluster extraction algorithm is used to extract the point cloud cluster of the target object from the first point cloud. Based on the fixed transformation matrix from the end effector to the camera and the transformation matrix from the robot base to the end effector, the position of the centroid of the target object in the reference coordinate system is calculated; The Euclidean distance between the gripper and the target object is calculated based on the position of the end effector and the position of the centroid.

6. The method according to claim 3, characterized in that, The method of classifying the vibration intensity of the vibration unit based on the wearer's perceptible vibration perception includes: The vibration intensity of the vibration unit is graded according to the working voltage range to obtain the initial level; The sensitivity of the ability to distinguish vibration stimuli was obtained through a perception threshold experiment. Clustering is performed based on the sensitivity of the ability to distinguish vibration stimuli, and the initial level is graded to obtain the vibration grade.

7. The method according to claim 6, characterized in that, The formula for calculating the sensitivity of the ability to distinguish vibration stimuli is as follows: in, The sensitivity to vibrational stimuli is represented by Hits, which is the number of times the stimulus intensity is correctly judged, and False Alarm refers to the number of times the stimulus intensity is incorrectly judged. Z represents the inverse function of the standard normal distribution.

8. The method according to claim 3, characterized in that, The clustering of sensitivity to vibration stimuli and the grading of the initial levels to obtain vibration grading include: Collect multiple d' results for each participant at different vibration intensities, and use the average of the d' results as the clustering input; The differences between data points are calculated using Euclidean distance, and a distance matrix is ​​constructed. The Ward method was selected as the clustering strategy, with the goal of minimizing the variance within groups. By merging samples step by step, a dendritic cluster diagram was formed, and the number of clusters was set based on this diagram and the shearing point, thus obtaining the clustering results. The members and characteristics of different clusters in the clustering results were analyzed to determine the vibration intensity boundary points that participants could distinguish, so that each cluster corresponds to a vibration level.

9. The method according to claim 3, characterized in that, Establish a mapping relationship between the distance between the gripper and the target object and the vibration classification, resulting in a second mapping relationship, including: The distance between the gripper and the target object is divided into distance levels that are the same as the vibration classification levels; By mapping distance levels to vibration classifications, a second mapping relationship is obtained.

10. A device for controlling a robotic arm to grasp enhanced kinematic imagination, characterized in that, include: The building module is used to construct motion image decoding models for recognizing grasping intentions in actual control applications; The first mapping module is used to establish the mapping relationship between the relative position of the gripper and the target object and the vibration area, and to obtain the first mapping relationship. The calculation module is used to calculate the distance between the mechanical gripper and the target object; The vibration grading module is used to grade the vibration intensity of the vibration unit based on the wearer's perceptible vibration sensation, thus obtaining a vibration grade. The second mapping module establishes a mapping relationship between the distance between the gripper and the target object and the vibration level, thus obtaining the second mapping relationship; The setting module is used to set the vibration intensity of each unit in the vibration module through the second mapping relationship; The encoding module is used to encode vibration feedback instructions based on the first and second mapping relationships, and to determine the activated finger vibration area to perform tactile feedback; The execution module is used to acquire the operator's motion imagery adjusted based on tactile feedback and decode the grasping intent in real time; If the intent exceeds the set threshold, a scraping operation will be performed.

Images