Robot autonomous grabbing method and system based on visual and tactile feedback

Abstract

The invention relates to a robot autonomous grabbing method and system based on visual and tactile feedback, and the method comprises the steps: firstly, observing deformation quantities of rubber at different joints of a manipulator in different directions through a camera, obtaining current manipulator deformation quantity data, splicing the current manipulator deformation quantity data with manipulator joint position data, and then carrying out global correlation analysis processing; obtaining a current manipulator global feature vector; then, the historical mechanical arm deformation quantity data and the current mechanical arm deformation quantity data are spliced and then subjected to time sequence feature analysis processing, and mechanical arm time sequence feature vectors are obtained; and finally, performing feature fusion on the current manipulator global feature vector and the manipulator time sequence feature vector, and performing nonlinear processing to obtain a manipulator deformation prediction vector. And the manipulator executes the grabbing action of the manipulator at the next moment according to the manipulator deformation prediction vector, the grabbing force of the manipulator is automatically adjusted, and the continuity and stability of the grabbing action of the manipulator are achieved.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a robot autonomous grasping method and system based on vision and tactile feedback. Background Art

[0002] A visual-tactile robot is a robot that combines visual and tactile capabilities. The robot's eyes (camera) can see the movements of the manipulator and interact through tactile perception. This allows it to not only perceive the environment through vision, but also perform precise operations through tactile feedback. It is widely used in robot tolerance testing, the combination of robots and prostheses, precision operation and assembly, and human-computer interaction.

[0003] Affected by the camera's visual perception problems, the manipulator's tactile perception problems, or the imperfect grasping pressure planning algorithm, the current vision-tactile robot has problems with complex calculations and incoherent grasping actions when performing grasping tasks, which affects the flexibility and accuracy of the vision-tactile robot's grasping actions. Summary of the invention

[0004] The purpose of the present invention is to provide a robot autonomous grasping method and system based on vision and tactile feedback.

[0005] The technical solution of the present invention is as follows:

[0006] A robot autonomous grasping method based on vision and tactile feedback includes the following operations:

[0007] S1. Based on the rubber images of different joints of the manipulator at the current moment obtained by the camera, the deformation variables in different directions of different joints of the manipulator at the current moment are obtained to form the current manipulator deformation variable data; the deformation variables in different directions of the joints include: positive and negative rubber offsets in the x-axis direction, contraction or expansion in the y-axis direction, and rubber stretching or compression in the z-axis direction; the current manipulator deformation variable data and the manipulator joint position data are spliced ​​to obtain the current manipulator deformation feature data; the current manipulator deformation feature data is processed by global correlation analysis to obtain the current manipulator global feature vector;

[0008] S2, obtaining the deformation variables of different joints of the manipulator in different directions at historical continuous moments, obtaining the manipulator deformation variable data at several historical moments, and forming the historical manipulator deformation variable data; the historical manipulator deformation variable data and the current manipulator deformation variable data are spliced ​​to obtain the manipulator comprehensive feature data; the manipulator comprehensive feature data is processed by time series feature analysis to obtain the manipulator time series feature vector;

[0009] S3. The current global feature vector of the manipulator and the temporal feature vector of the manipulator are fused to obtain a manipulator fusion vector; the manipulator fusion vector is processed nonlinearly to obtain a manipulator deformation prediction vector; the manipulator executes the manipulator grasping action at the next moment according to the manipulator deformation prediction vector.

[0010] The operation of global correlation analysis processing in S1 is realized through self-attention processing, feedforward neural network processing, layer normalization and residual connection processing.

[0011] The specific operations of the temporal feature analysis and processing in S2 are as follows: the manipulator comprehensive feature data is processed by different linear layers to obtain a manipulator comprehensive linear query vector, a manipulator comprehensive linear key vector and a manipulator comprehensive linear value vector; the manipulator comprehensive linear query vector, the manipulator comprehensive linear key vector and the manipulator comprehensive linear value vector are respectively decomposed into several sub-quantities to obtain several manipulator comprehensive linear query sub-vectors, several manipulator comprehensive linear key sub-vectors and several manipulator comprehensive linear value sub-vectors; the attention scores of each manipulator comprehensive linear query sub-vector and each manipulator comprehensive linear key sub-vector are obtained, and the scores are added to the corresponding mask matrices and processed by probability mapping to obtain several mask attention scores; several mask attention scores are weightedly summed with each manipulator comprehensive linear value sub-vector to obtain several initial output quantities; after the several initial output quantities are spliced, they are processed by linear processing and feedforward neural network to obtain the manipulator temporal feature vector.

[0012] The deformation amounts in different directions of different joints of the manipulator at the current moment in S1 are obtained based on the rubber pixel distribution information in the rubber image of different joints of the manipulator at the current moment and the rubber pixel distribution information in the natural state rubber image of the corresponding joints.

[0013] The deformation amount in the x-axis, y-axis and z-axis directions of the current joint connection is the product of the maximum absolute value of the coordinates of the rubber pixel points in the rubber image of the current joint connection on the x-axis, y-axis and z-axis, and the maximum absolute value of the coordinates of the rubber pixel points in the natural state rubber image of the current joint connection on the corresponding coordinate axis, respectively, and the image scale.

[0014] The feature fusion operation in S3 can be achieved through multi-head attention processing. In the multi-head attention processing, the robot query vector is obtained based on the robot comprehensive feature data, and the robot value vector and robot key vector are obtained based on the robot time series feature vector.

[0015] A robot system based on vision and tactile feedback, used to implement the above-mentioned robot autonomous grasping method based on vision and tactile feedback, comprising:

[0016] A camera, used to acquire images;

[0017] The manipulator, including joints and rubber at joint connections, is used to perform a grasping action according to a manipulator action execution signal transmitted from a control unit;

[0018] The calculation unit is used to obtain the deformation variables in different directions of different joint connections of the manipulator at the current moment based on the rubber image of the manipulator at different joint connections obtained by the camera, so as to form the current manipulator shape variable data; the current manipulator shape variable data and the manipulator joint position data are spliced ​​to obtain the current manipulator deformation feature data; the current manipulator deformation feature data is processed by global correlation analysis to obtain the current manipulator global feature vector; it is used to obtain the deformation variables in different directions of different joint connections of the manipulator at historical continuous moments, obtain the manipulator shape variable data at several historical moments, and form the historical manipulator shape variable data; the historical manipulator shape variable data and the current manipulator shape variable data are spliced ​​to obtain the manipulator comprehensive feature data; the manipulator comprehensive feature data is processed by time series feature analysis to obtain the manipulator time series feature vector; the current manipulator global feature vector and the manipulator time series feature vector are used for feature fusion to obtain the manipulator fusion vector; the manipulator fusion vector is processed by nonlinearity to obtain the manipulator deformation prediction vector;

[0019] The control unit is used to decompose the robot deformation prediction vector transmitted from the calculation unit into a robot action execution signal and transmit it to the robot.

[0020] The beneficial effects of the present invention are:

[0021] The present invention provides a robot autonomous grasping method based on vision and tactile feedback. First, a camera is used to obtain rubber images at different joint connections of a manipulator at the current moment, and the deformation variables of the rubber at different joint connections of the manipulator at the current moment in different directions are obtained after analyzing the deformation variables of the rubber, thereby forming the deformation variable data of the current manipulator; the current manipulator deformation variable data and the manipulator joint position data are spliced ​​to obtain the current manipulator deformation feature data containing the manipulator joint position information and the applied pressure information; the current manipulator deformation feature data is subjected to global correlation analysis and processing to capture the global correlation of the joint connection to obtain the current manipulator global feature vector; then, the deformation variables at different directions of different joint connections of the manipulator at historical continuous moments are obtained to obtain the historical information that can reflect the pressure and deformation information when the manipulator joint performs historical actions. Manipulator shape variable data; splice the historical manipulator shape variable data with the current manipulator shape variable data to obtain the manipulator comprehensive feature data; the manipulator comprehensive feature data is processed by time series feature analysis, focusing on the dynamic pressure and deformation change characteristics of the manipulator action, and obtaining the manipulator time series feature vector; finally, after feature fusion of the current manipulator global feature vector and the manipulator time series feature vector, the manipulator fusion vector is obtained while analyzing the current manipulator action pressure and combining the historical manipulator action data; the manipulator fusion vector is processed nonlinearly to obtain the manipulator deformation prediction vector; the manipulator executes the manipulator grasping action at the next moment according to the manipulator deformation prediction vector, and autonomously adjusts the manipulator's grasping force to ensure that it can firmly grasp the object and avoid damage caused by excessive force, so as to achieve the continuity of the manipulator grasping action;

[0022] The present invention provides a robot autonomous grasping method based on vision and tactile feedback. The method observes the deformation characteristics of the manipulator through a camera, analyzes the deformation characteristics of the manipulator at the current moment, refers to the deformation information of the manipulator action at the historical moment, performs multi-moment information fusion, predicts the manipulator operation strategy at the next moment, and realizes efficient, stable and coherent grasping.

[0023] The present invention provides a robot autonomous grasping method based on vision and tactile feedback, which has stable motion planning effect and can be applied to a variety of complex scenes. Regardless of whether the target object has a regular shape or a hard or soft material, the grasping method can be intelligently adjusted to complete the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] By reading the detailed description of the preferred embodiment below, the scheme and advantages of the present application will become clear to those skilled in the art. The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present invention.

[0025] In the attached picture:

[0026] Figure 1Schematic diagram of the process of the robot autonomous grasping method in the embodiment. DETAILED DESCRIPTION

[0027] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0028] This embodiment provides a robot system based on vision and tactile feedback, which is used to implement a robot autonomous grasping method based on vision and tactile feedback, including:

[0029] A camera, used to acquire images;

[0030] The manipulator, including joints and rubber at joint connections, is used to perform a grasping action according to a manipulator action execution signal transmitted from a control unit;

[0031] The calculation unit is used to obtain the deformation variables in different directions of different joint connections of the manipulator at the current moment based on the rubber image of the manipulator at different joint connections obtained by the camera, so as to form the current manipulator shape variable data; the current manipulator shape variable data and the manipulator joint position data are spliced ​​to obtain the current manipulator deformation feature data; the current manipulator deformation feature data is processed by global correlation analysis to obtain the current manipulator global feature vector; it is used to obtain the deformation variables in different directions of different joint connections of the manipulator at historical continuous moments, obtain the manipulator shape variable data at several historical moments, and form the historical manipulator shape variable data; the historical manipulator shape variable data and the current manipulator shape variable data are spliced ​​to obtain the manipulator comprehensive feature data; the manipulator comprehensive feature data is processed by time series feature analysis to obtain the manipulator time series feature vector; the current manipulator global feature vector and the manipulator time series feature vector are used for feature fusion to obtain the manipulator fusion vector; the manipulator fusion vector is processed by nonlinearity to obtain the manipulator deformation prediction vector;

[0032] The control unit is used to decompose the robot deformation prediction vector transmitted from the calculation unit into a robot action execution signal and transmit it to the robot.

[0033] A robot autonomous grasping method based on vision and tactile feedback, see Figure 1 , the specific steps are as follows.

[0034] S1. Based on the camera, the rubber images of different joints of the manipulator at the current moment are obtained to obtain the deformation variables in different directions of different joints of the manipulator at the current moment, thereby forming the current manipulator shape variable data; the current manipulator shape variable data and the manipulator joint position data are spliced ​​to obtain the current manipulator deformation feature data; the current manipulator deformation feature data is processed by global correlation analysis to obtain the current manipulator global feature vector.

[0035] A camera is used to obtain rubber images at different joints of the manipulator at the current moment. By observing the deformation of the rubber at the joints of the manipulator, the deformation of different joints of the manipulator in different directions at the current moment is obtained to form the current manipulator shape variable data; the current manipulator shape variable data and the manipulator joint position data are spliced ​​to obtain the current manipulator deformation feature data containing the manipulator joint position information and the applied pressure information; the current manipulator deformation feature data is subjected to global correlation analysis and processing to capture the global correlation between the joint connections to obtain the current manipulator global feature vector.

[0036] Firstly, based on the rubber images of different joints of the manipulator at the current moment acquired by the camera, the deformation quantities of different joints of the manipulator in different directions at the current moment are obtained.

[0037] Among them, the deformation amounts in different directions of different joint connections of the manipulator at the current moment are obtained based on the rubber pixel distribution information in the rubber image of different joint connections of the manipulator at the current moment and the rubber pixel distribution information in the natural state rubber image of the corresponding joint connections.

[0038] For example, there are 14 joints, which are the finger joints of the five fingers of the robot. Each of the above joints has a total of 6 deformation variables in different directions, including: positive and negative rubber offset in the x-axis direction, contraction or expansion in the y-axis direction, and rubber stretching or compression in the z-axis direction.

[0039] Taking the current joint connection as an example, the deformation of the current joint connection of the manipulator in the x-axis, y-axis and z-axis directions at the current moment is the product of the maximum absolute value of the coordinates of the rubber pixel points in the rubber image of the current joint connection on the x-axis, y-axis and z-axis, and the maximum absolute value of the coordinates of the rubber pixel points in the natural state rubber image of the current joint connection on the corresponding coordinate axis, respectively, and the image scale.

[0040] Then, based on the deformation variables in different directions of different joint connections of the manipulator at the current moment, the current manipulator deformation variable data in the form of a 14×6 matrix is ​​formed.

[0041] Next, the current manipulator deformation feature data and the manipulator joint position data (a matrix formed by the position information of all joint connections of the manipulator) are spliced ​​to obtain the current manipulator deformation feature data containing the joint position information.

[0042] Finally, the deformation feature data of the current manipulator is processed by global correlation analysis to obtain the global feature vector of the current manipulator. The global correlation analysis operation is realized by self-attention processing, feedforward neural network processing, layer normalization and residual connection processing.

[0043] S2. Obtain the deformation variables in different directions of different joint connections of the manipulator at historical continuous moments to obtain historical manipulator shape variable data; concatenate the historical manipulator shape variable data with the current manipulator shape variable data to obtain comprehensive feature data of the manipulator; and process the comprehensive feature data of the manipulator through time series feature analysis to obtain a time series feature vector of the manipulator.

[0044] The deformation variables in different directions of different joint connections of the manipulator at historical continuous moments are obtained to obtain historical manipulator shape variable data that can reflect the pressure deformation information of the manipulator joints when performing historical actions; the historical manipulator shape variable data is spliced ​​with the current manipulator shape variable data to obtain the manipulator comprehensive feature data; the manipulator comprehensive feature data is processed by time series feature analysis, focusing on the dynamic pressure and deformation change characteristics of the manipulator action, and obtaining the manipulator time series feature vector.

[0045] First, the deformation variables in different directions of different joint connections of the manipulator at historical continuous moments (preferably the 9 historical moments before the current moment) are obtained to obtain the manipulator shape variable data at several historical moments, thus forming the historical manipulator shape variable data.

[0046] Then, the historical manipulator shape variable data and the current manipulator shape variable data are spliced ​​(they can be spliced ​​after being converted into vector forms respectively) to obtain the manipulator comprehensive characteristic data reflecting the manipulator timing information.

[0047] Finally, the comprehensive feature data of the manipulator is processed by time series feature analysis, focusing on the dynamic change characteristics of the manipulator's movements to obtain the manipulator's time series feature vector.

[0048] The operation steps of the above-mentioned timing feature analysis processing are as follows.

[0049] Step 1: The manipulator comprehensive feature data is processed by different linear layers (which can be achieved by multiplying the manipulator comprehensive feature data in matrix form with the query item parameter matrix, key item parameter matrix and value item parameter matrix respectively) to obtain the manipulator comprehensive linear query vector, manipulator comprehensive linear key vector and manipulator comprehensive linear value vector.

[0050] Step 2: Decompose the manipulator comprehensive linear query vector, manipulator comprehensive linear key vector and manipulator comprehensive linear value vector into several sub-quantities (this can be achieved by multiplying the manipulator comprehensive linear query vector, manipulator comprehensive linear key vector and manipulator comprehensive linear value vector with different parameter weights respectively), and obtain several manipulator comprehensive linear query sub-vectors, several manipulator comprehensive linear key sub-vectors and several manipulator comprehensive linear value sub-vectors.

[0051] Step 3: Obtain the attention scores of each manipulator integrated linear query subvector and each manipulator integrated linear key subvector, add them to the corresponding mask matrix and process them through probability mapping to obtain several mask attention scores. The attention score is obtained based on the product of the manipulator integrated linear query subvector and the manipulator integrated linear key subvector.

[0052] Step 4: Several masked attention scores are weightedly summed with each manipulator's comprehensive linear value sub-vector to obtain several initial output quantities; after several initial output quantities are spliced, they are processed by linear processing and feedforward neural network to obtain the manipulator's temporal feature vector.

[0053] S3. The current global feature vector of the manipulator and the temporal feature vector of the manipulator are fused to obtain a manipulator fusion vector; the manipulator fusion vector is processed nonlinearly to obtain a manipulator deformation prediction vector; the manipulator executes the manipulator grasping action at the next moment according to the manipulator deformation prediction vector.

[0054] After feature fusion of the current manipulator global feature vector and the manipulator temporal feature vector, the historical manipulator motion data is combined with the current manipulator motion pressure to achieve the fusion of multi-time manipulator deformation information and obtain the manipulator fusion vector; the manipulator fusion vector is processed nonlinearly to obtain the manipulator deformation prediction vector; the manipulator executes the manipulator grasping action at the next moment according to the manipulator deformation prediction vector.

[0055] First, the current manipulator global feature vector and the manipulator temporal feature vector are feature fused to obtain a manipulator fusion vector containing the manipulator feature information at the current moment and the manipulator feature information at the historical moment. The feature fusion operation can be achieved through multi-head attention processing. In the multi-head attention processing, the manipulator query vector is obtained based on the manipulator comprehensive feature data, and the manipulator value vector and manipulator key vector are obtained based on the manipulator temporal feature vector.

[0056] Then, the manipulator fusion vector is processed nonlinearly to obtain the manipulator deformation prediction vector. The nonlinear processing operation can be achieved by processing the manipulator fusion vector through the trained neural network layer.

[0057] Finally, the manipulator executes the manipulator grasping action at the next moment according to the predicted rubber deformation at each joint connection in the manipulator deformation prediction vector, and autonomously adjusts the manipulator's grasping force to ensure that it can firmly grasp the object while avoiding damage caused by excessive force, thereby achieving the continuity of the manipulator's grasping action.

[0058] The present embodiment provides a robot autonomous grasping method based on vision and tactile feedback. First, a camera is used to obtain rubber images of different joint connections of a manipulator at the current moment. After analyzing the rubber deformation, deformations in different directions of different joint connections of the manipulator at the current moment are obtained to form the current manipulator deformation data. The current manipulator deformation data and the manipulator joint position data are spliced ​​to obtain the current manipulator deformation feature data containing the manipulator joint position information and the applied pressure information. The current manipulator deformation feature data is subjected to global correlation analysis and processing to capture the global correlation of the joint connection to obtain the current manipulator global feature vector. Then, deformations in different directions of different joint connections of the manipulator at historical continuous moments are obtained to obtain historical information that can reflect the pressure and deformation information of the manipulator joints when performing historical actions. Historical manipulator shape variable data; splicing the historical manipulator shape variable data with the current manipulator shape variable data to obtain the manipulator comprehensive feature data; the manipulator comprehensive feature data is processed by time series feature analysis, focusing on the dynamic pressure and deformation change characteristics of the manipulator action, and obtaining the manipulator time series feature vector; finally, after feature fusion of the current manipulator global feature vector and the manipulator time series feature vector, while analyzing the current manipulator action pressure and combining the historical manipulator action data, the manipulator fusion vector is obtained; the manipulator fusion vector is processed nonlinearly to obtain the manipulator deformation prediction vector; the manipulator executes the manipulator grasping action at the next moment according to the manipulator deformation prediction vector, and autonomously adjusts the manipulator's grasping force to ensure that it can not only firmly grasp the object, but also avoid damage caused by excessive force, so as to achieve the continuity of the manipulator grasping action;

[0059] This embodiment provides a robot autonomous grasping method based on vision and tactile feedback. It observes the deformation characteristics of the manipulator through a camera, analyzes the deformation characteristics of the manipulator at the current moment, refers to the deformation information of the manipulator's movements at historical moments, performs multi-moment information fusion, predicts the manipulator's operation strategy at the next moment, and achieves efficient, stable, and coherent grasping.

[0060] The present embodiment provides a robot autonomous grasping method based on vision and tactile feedback, which has stable motion planning effect and can be applied to a variety of complex scenarios. Regardless of whether the target object has a regular shape or a hard or soft material, the grasping method can be intelligently adjusted to complete the task.

Claims

1. A robot autonomous grasping method based on vision and tactile feedback, characterized in that: The following operations are included: S1, based on the rubber images of different joints of the manipulator at the current moment obtained by the camera, the deformation variables of different joints of the manipulator at the current moment in different directions are obtained to form the deformation variable data of the current manipulator; The deformation variables in different directions of the joint connection include: positive and negative rubber offset in the x-axis direction, contraction or expansion in the y-axis direction, and rubber stretching or compression in the z-axis direction; The current manipulator shape variable data and the manipulator joint position data are spliced ​​to obtain the current manipulator deformation feature data; The deformation feature data of the current manipulator is processed by global correlation analysis to obtain the global feature vector of the current manipulator; S2, obtaining the deformation variables of different joints of the manipulator in different directions at the historical continuous moments, obtaining the manipulator deformation variable data at several historical moments, and forming the historical manipulator deformation variable data; the historical manipulator deformation variable data and the current manipulator deformation variable data are spliced ​​to obtain the manipulator comprehensive feature data; The comprehensive characteristic data of the manipulator is processed by time series characteristic analysis to obtain the manipulator time series characteristic vector; S3, the current manipulator global feature vector and the manipulator temporal feature vector are subjected to feature fusion to obtain a manipulator fusion vector; the manipulator fusion vector is subjected to nonlinear processing to obtain a manipulator deformation prediction vector; The manipulator performs the manipulator grasping action at the next moment according to the manipulator deformation prediction vector.

2. The robot autonomous grasping method based on vision and tactile feedback according to claim 1 is characterized in that: The operation of global correlation analysis processing in S1 is achieved through self-attention processing, feedforward neural network processing, layer normalization and residual connection processing.

3. The robot autonomous grasping method based on vision and tactile feedback according to claim 1 is characterized in that: In S2, the operations of the time series feature analysis and processing are specifically as follows: The manipulator comprehensive feature data is processed by different linear layers to obtain a manipulator comprehensive linear query vector, a manipulator comprehensive linear key vector and a manipulator comprehensive linear value vector; Decomposing the manipulator integrated linear query vector, the manipulator integrated linear key vector and the manipulator integrated linear value vector into a plurality of sub-quantities respectively, to obtain a plurality of manipulator integrated linear query sub-vectors, a plurality of manipulator integrated linear key sub-vectors and a plurality of manipulator integrated linear value sub-vectors; Obtain the attention scores of each manipulator comprehensive linear query sub-vector and each manipulator comprehensive linear key sub-vector, add them to the corresponding mask matrix respectively, and then process them through probability mapping to obtain several mask attention scores; Several mask attention scores are weighted and summed with each manipulator's comprehensive linear value sub-vector to obtain several initial output quantities; After several initial output quantities are spliced, processed by linear processing and feedforward neural network, the manipulator timing feature vector is obtained.

4. The robot autonomous grasping method based on vision and tactile feedback according to claim 1 is characterized in that: In S1, the deformation amounts in different directions at different joints of the manipulator at the current moment are obtained based on the rubber pixel distribution information in the rubber images at different joints of the manipulator at the current moment and the rubber pixel distribution information in the natural state rubber images of the corresponding joints.

5. The robot autonomous grasping method based on vision and tactile feedback according to claim 4 is characterized in that: The deformation amount in the x-axis, y-axis and z-axis directions of the current joint connection is the product of the maximum absolute value of the coordinates of the rubber pixel points in the rubber image of the current joint connection on the x-axis, y-axis and z-axis, and the maximum absolute value of the coordinates of the rubber pixel points in the natural state rubber image of the current joint connection on the corresponding coordinate axis, respectively, and the image scale.

6. The robot autonomous grasping method based on vision and tactile feedback according to claim 1 is characterized in that: In S3, the feature fusion operation can be implemented through multi-head attention processing. In the multi-head attention processing, the manipulator query vector is obtained based on the manipulator comprehensive feature data, and the manipulator value vector and the manipulator key vector are obtained based on the manipulator time series feature vector.

7. A robot system based on vision and tactile feedback, used to implement the robot autonomous grasping method based on vision and tactile feedback as claimed in claim 1, characterized in that it includes: A camera, used to acquire images; The manipulator, including joints and rubber at joint connections, is used to perform a grasping action according to a manipulator action execution signal transmitted from a control unit; The calculation unit is used to obtain the deformation variables in different directions of the different joint connections of the manipulator at the current moment based on the rubber images of the different joint connections of the manipulator at the current moment obtained by the camera, so as to form the deformation variable data of the current manipulator; the deformation variable data of the current manipulator and the joint position data of the manipulator are spliced ​​to obtain the deformation feature data of the current manipulator; the deformation feature data of the current manipulator is processed by global correlation analysis to obtain the global feature vector of the current manipulator; It is used to obtain the deformation variables of different joints of the manipulator in different directions at historical continuous moments, obtain the manipulator shape variable data at several historical moments, and form the historical manipulator shape variable data; the historical manipulator shape variable data and the current manipulator shape variable data are spliced ​​to obtain the manipulator comprehensive feature data; the manipulator comprehensive feature data is processed by time series feature analysis to obtain the manipulator time series feature vector; The global feature vector of the current manipulator and the temporal feature vector of the manipulator are used to obtain the manipulator fusion vector through feature fusion; the manipulator fusion vector is processed nonlinearly to obtain the manipulator deformation prediction vector; The control unit is used to decompose the robot deformation prediction vector transmitted from the calculation unit into a robot action execution signal and transmit it to the robot.

Images