Teleoperation robot auxiliary system and method based on vision and imitation learning

Abstract

The invention discloses a teleoperation robot auxiliary system and method based on vision and imitation learning, the system comprises a teleoperation module, a skill learning module, a vision processing module and a virtual force auxiliary module, the teleoperation module realizes remote operation of a robot through an external control device; the skill learning module learns an expert operation track through a dynamic motion primitive algorithm to generate a generalization reference path, the visual processing module combines a three-dimensional target positioning technology of a Kinect depth camera to realize environment perception, and the virtual force auxiliary module constructs a self-adaptive virtual clamp force field based on a position error. A force selector mechanism is employed to dynamically adjust the haptic feedback intensity, guide force is applied to correct when the operation deviates from a reference trajectory, and the operation autonomy is maintained when the trajectory is fitted. Through the synergistic effect of expert track guidance and a visual constraint force field, the operation precision is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of teleoperation robots, and in particular to a teleoperation robot auxiliary system and method based on vision and imitation learning. Background Art

[0002] With the continuous development of robotics technology, especially in the fields of industrial automation, medical assistance and dangerous operations, teleoperated robots are gradually becoming a key tool to replace manual labor. In many complex and high-risk operating environments, teleoperation systems can provide operators with the ability to remotely operate and complete tasks that are high-precision and dangerous. However, despite the significant advantages of teleoperation systems, traditional teleoperation methods still have many problems, especially in terms of accuracy, operating habits, environmental changes, etc. Traditional teleoperation methods usually rely on physical input devices such as joysticks and buttons. When facing complex tasks, such devices are easily affected by the operator's hand shake, operating errors and environmental interference, which leads to low system control accuracy and high difficulty in operation. Operators need to undergo a long period of professional training to master them. This makes it difficult for teleoperation systems to fully exert their advantages in high-precision tasks or dynamic environments.

[0003] At present, the traditional teleoperation system has problems such as low precision, unskilled operators, hand tremors and environmental interference. Due to the operator's operating skills and environmental complexity, it is difficult for the traditional teleoperation system to meet the requirements of high precision and high efficiency. Especially in dynamic and uncertain environments, the control performance of the teleoperation system is often greatly limited. Hand tremors are a significant problem, which not only affects the operator's operating accuracy, but also may cause unnecessary collisions between the robot and the environment, increasing the risk of the task. At the same time, without professional training, the operator's operation stability is poor, which is prone to large errors, resulting in inaccurate or delayed task completion.

[0004] For example, the robot teleoperation assistance system based on binocular stereo vision with patent publication number CN107911687A realizes the target posture calculation and stereoscopic display function through the coordinated processing of the measurement camera and the monitoring camera. Although this system can improve the accuracy of posture measurement, it does not introduce the operator behavior imitation learning mechanism, and does not design a filtering algorithm for hand tremors. It is easily affected by the difference in operator skills in a dynamic environment, resulting in the accumulation of end effector jitter errors.

[0005] For example, the master-slave teleoperated robot system and method with patent publication number CN116572217A assists operators in completing complex tasks through virtual fixtures. The system combines force feedback and motion control to improve operational accuracy, but does not introduce visual information to assist in environmental modeling, and has limited adaptability in dynamic environments.

[0006] For example, the robot teleoperation system and method based on visual positioning and virtual reality technology with patent publication number CN117584123A generates three-dimensional point cloud information of the remote robot working scene through a depth camera and a scene reconstruction system, and realizes precise control of the virtual robot posture in combination with virtual reality equipment. This system can significantly improve the operator's ability to perceive the environment, but does not filter the operator's hand tremors and requires high computing resources. Summary of the invention

[0007] In order to overcome the defects and shortcomings of the prior art, the present invention provides a remote-controlled robot assistance system and method based on vision and imitation learning. The present invention learns the expert operation trajectory and establishes a skill model through a dynamic motion primitive algorithm, and generates a reference trajectory with generalization capability. During the remote operation process, a depth camera is used to obtain the three-dimensional coordinates of the target object, and the position error is generated by combining the real-time posture of the robot with the reference trajectory; a virtual fixture force is constructed based on vision, and the position error is converted into a tactile guiding force and fed back to the operating end device; at the same time, the virtual fixture parameters are dynamically adjusted in real time. When an environmental obstacle is detected, the reference trajectory topology is reconstructed according to the visual information to guide the operator to complete the obstacle avoidance operation. The present invention can effectively improve the control accuracy of novice operators, reduce the operating load, and shorten the task completion time.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] The present invention provides a teleoperation robot assistance system based on vision and imitation learning, comprising: a teleoperation module, a skill learning module, a visual processing module and a virtual force assistance module;

[0010] The teleoperation module realizes remote operation of the robot through an external control device, establishes a two-way communication link with the slave robot, pairs the control space of the external control device with the workspace of the robot through the master-slave space matching mechanism, and adjusts the control response of the robot by adjusting the proportional factor;

[0011] The skill learning module is used to extract and learn the features of the expert trajectory, build an expert skill library based on the improved dynamic motion primitive algorithm, and generate the best reference trajectory, combined with the real-time status of the robot, to provide guidance for the teleoperation module;

[0012] The visual processing module generates a depth map through a stereo vision algorithm, establishes a transformation matrix from the camera coordinate system to the robot base coordinate system, performs target recognition through HSV color space conversion and morphological processing, calculates the three-dimensional coordinates of the target object in the camera coordinate system based on the depth map, and converts it into a posture in the robot coordinate system through a pre-calibrated transformation matrix;

[0013] The virtual force auxiliary module obtains the robot state and the optimal reference trajectory, searches for the optimal target point of the robot on the optimal reference trajectory, constructs the gravitational force of the optimal target point and the task completion point on the end of the robot, and feeds back to the teleoperation module.

[0014] As a preferred technical solution, there is provided the above-mentioned teleoperation robot assistance system based on vision and imitation learning, comprising the following steps:

[0015] Based on the teleoperation module, the control space of the external control device is paired with the workspace of the robot, and the coordinate systems of the master and slave robots are calibrated;

[0016] Based on the visual processing module, the real-time posture mapping of the master and slave robots is performed, the real-time posture of the tactile device is collected, the target posture of the slave end is calculated, and the joint angle is solved through inverse kinematics and sent to the slave end robot arm;

[0017] Model and generalize the expert operation trajectory based on the skill learning module, build the expert skill library based on the improved dynamic motion primitive algorithm, and generate the best reference trajectory;

[0018] Capture remote target information based on Kinect;

[0019] Extracting the location of objects based on image processing;

[0020] Obtain the slave robot posture and expert trajectory, calculate the point on the expert trajectory that is closest to the current slave robot posture and calculate the distance;

[0021] The virtual fixture guiding force is calculated based on the virtual force auxiliary module.

[0022] As a preferred technical solution, the control space of the external control device is paired with the workspace of the robot based on the teleoperation module, and the coordinate system of the master and slave robots is calibrated, which specifically includes:

[0023] Define the initial pose X of the master tactile device l0 =[x l0 ,y l0 , z l0 ] and the initial position X of the slave robot arm f0 =[x f0 ,y f0 , z f0 ], based on the preset scaling matrix S = diag(S x , S y , S z ), so that the master-slave displacement satisfies X f =X f0 +S·(X l -X l0 ).

[0024] As a preferred technical solution, the real-time posture mapping of the master and slave robots is performed based on the visual processing module, and the real-time posture of the tactile device is collected, which is specifically expressed as follows:

[0025] X l (t) = [x l (t),y l (t),z l (t)]

[0026] Calculate the target pose from the end:

[0027]

[0028] As a preferred technical solution, the expert operation trajectory is modeled and generalized based on the skill learning module, which specifically includes:

[0029] Obtain the expert operation trajectory and record the end-to-end trajectory sequence Contains position x, speed Acceleration By building a second-order differential equation, x, and The relationship is as follows:

[0030]

[0031] Among them, K l is the stiffness matrix, D l is the damping matrix, τ is the time scaling factor, f(z) is the nonlinear term, and the phase variable z is then defined as:

[0032]

[0033] Among them, α is the attenuation coefficient;

[0034] The nonlinear term f(z) is defined using the Gaussian basis function as follows:

[0035]

[0036] Among them, c i is the basis function center, σ i is the width, w i is the weight coefficient;

[0037] For position x, and The demonstrated trajectory gives the objective function:

[0038]

[0039] As a preferred technical solution, capturing remote target information based on Kinect specifically includes:

[0040] Set the position of the remote robot end effector to x f =[x r ,y r ,z r ], the corresponding position in Kinect is x c =[x c ,y c ,z c ];

[0041] x f and x c The relationship is defined as follows:

[0042]

[0043] Using LSM algorithm for calibration:

[0044]

[0045] Among them, I 4 is the identity matrix, [x ri ,y ri ,z ri ] i＝n , and [x ci ,y ci ,z ci ] are the positions of the i-th point in the remote robot coordinate system and the Kinect coordinate system, respectively.

[0046] Based on the calibration matrix T, the position in the image corresponds to the position of the robot coordinate system:

[0047]

[0048] As a preferred technical solution, extracting the position of an object based on image processing specifically includes:

[0049] Get the two-dimensional image information of Kinect, extract the HSV three-channel image and perform binarization processing, perform corrosion and expansion operations on the binary image, take the maximum connected area of ​​the three binary maps, get the rectangular shape, and get the square area nodes, add the object depth information to the square area nodes, extract the three-dimensional position in the camera coordinate system, and calculate the position in the remote robot coordinate system.

[0050] As a preferred technical solution, the virtual fixture guiding force is calculated based on the virtual force auxiliary module, which is specifically expressed as:

[0051] F guidance =K vf1 X error (t)+K vf2 (Xvr -X f (t))

[0052] X error (t) = X gen (t closest )-X f (t)

[0053] Among them, F guidance represents the virtual fixture guiding force, X f (t) represents the position of the slave robot, X gen (t closest ) represents the point on the expert trajectory that is closest to the current slave robot posture, K vf1 is the expert trajectory tracking gain, K vf2 Attracts buffs to mission objectives.

[0054] As a preferred technical solution, the virtual fixture force fed back by the master-end tactile device is set according to the force selector, which is specifically expressed as:

[0055]

[0056] Among them, X error It represents the distance of the point on the expert trajectory that is closest to the current slave robot posture, and ∈ represents the set threshold.

[0057] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0058] (1) The present invention is based on robot learning of dynamic motion primitives, learns task trajectories from expert operations and generates a generalized trajectory model, providing expert-level reference paths for novice operators, effectively shortening the skill training cycle and improving operation standardization.

[0059] (2) The present invention combines visual recognition and virtual fixture technology to obtain the three-dimensional coordinates of the target object in real time through a depth camera, and dynamically generates force guidance feedback based on position errors to form a virtual gravitational field with spatial constraints, which can effectively reduce the trajectory error of novice operations.

[0060] (3) The present invention proposes a force selector mechanism that intelligently switches the guidance mode according to the task stage: providing strong guidance force to correct the operation when it deviates from the expert trajectory, and maintaining operational autonomy when the trajectory matches. This adaptive adjustment enables the system to significantly reduce the operator's cognitive load while improving the efficiency of task completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The overall architecture diagram of the teleoperated robot assistance system based on vision and imitation learning of the present invention;

[0062] Figure 2It is a schematic diagram of the virtual fixture force of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0064] Example 1

[0065] like Figure 1 As shown, this embodiment provides a teleoperation robot assistance system based on vision and imitation learning, including: a teleoperation module, a skill learning module, a visual processing module and a virtual force assistance module;

[0066] Among them, the teleoperation module is used to realize remote operation of the robot through an external control device, so that the robot can perform tasks in high-risk or human-inaccessible environments. The core of this module is constructed through a force feedback tactile device to ensure that the operator can accurately control the robot; first, the teleoperation module uses a six-degree-of-freedom force feedback device TouchX as the master controller, and a built-in high-precision encoder collects the operator's posture data in real time. A two-way communication link is established with the slave robot through the TCP / IP protocol. Through the matching mechanism of the master-slave space, the control space of TouchX is effectively paired with the robot's workspace to ensure that the movement of the teleoperation module is accurately synchronized with the movement of the robot's end, so that the operator can control the robot within a reasonable range. Secondly, by adjusting the proportional factor, the operator's control response to the robot is optimized, making the teleoperation process smoother and more sensitive, thereby ensuring that when performing tasks, the operator can interact with the robot efficiently and maintain good tracking performance. The maximum output force of the six-degree-of-freedom force feedback device TouchX in this embodiment is 10N, the position resolution is 0.01mm, and the force refresh frequency is not less than 1kHz, which meets the needs of precision teleoperation robots;

[0067] The skill learning module is used to analyze, model and generalize the collected expert trajectories. By extracting and learning the features of the task trajectories completed by the expert operation, the task-related mathematical model is established. When an untrained operator performs a teleoperation task, the optimal reference trajectory is automatically generated according to the task requirements, and combined with the real-time status of the robot, guidance is provided on the teleoperation device. In this way, the operator can complete the task more smoothly and accurately with the help of the system.

[0068] The skill learning module builds an expert skill library based on the improved dynamic motion primitive algorithm. The dynamic motion primitive can encode multi-dimensional teaching trajectory data and generate a generalized trajectory template containing position, velocity, and acceleration information.

[0069] The visual processing module is used to provide accurate spatial reference for the force guidance of the virtual fixture through environmental perception and target positioning technology. The core of this module is built through multi-sensor fusion to ensure that the operator can accurately identify the work object. First, the visual processing module uses the Kinect v2 depth camera as the main sensor. Its RGB camera works with the infrared depth sensor to generate a depth map through a stereo vision algorithm. By establishing a 4×4 transformation matrix from the camera coordinate system to the robot base coordinate system, the least squares method is used to optimize the parameters of 20 sets of calibration point data to achieve accurate mapping of three-dimensional space coordinates, and the mapping error is controlled within the range of ±1.5mm. Secondly, target recognition is achieved through HSV color space conversion and morphological processing: the RGB image is converted into HSV channels, and the target area is extracted based on threshold segmentation; the binary image is optimized through corrosion and expansion operations, the maximum connected domain is detected and the circumscribed rectangle is fitted; the three-dimensional coordinates of the target object in the camera coordinate system are calculated in combination with the depth information, and finally converted to the pose in the robot coordinate system through the pre-calibrated transformation matrix.

[0070] In this embodiment, the depth camera is a Kinect v2 device, whose RGB camera resolution is 1920×1080, the depth sensor accuracy reaches 0.5mm*1m, and the frame rate is stable at 30fps, which meets the visual perception requirements of scenes such as precision assembly.

[0071] The virtual fixture module is used to provide dynamic auxiliary guidance for remote operation in complex environments through multimodal perception and adaptive force field coupling technology. This module is built based on the collaborative mechanism of expert trajectory learning, obstacle avoidance and operator intention recognition. First, the robot state and optimal reference trajectory are obtained in real time, and the current optimal target point of the robot is searched on the optimal reference trajectory. The gravitational force of the optimal target point and the task completion point on the end of the robot is constructed and fed back to the remote operation device to guide the operator to remotely control the robot.

[0072] The virtual fixture module contains multiple dynamically adjustable parameters, including the gravitational weight of the task completion point on the robot, the gravitational weight of the trajectory optimal point on the robot, etc.

[0073] In this embodiment, the expert completes 3-5 standard operations to build an expert trajectory library, records the trajectory data and trains the skill learning module. Then, when the novice performs the remote operation task, the task target position information is obtained through the visual module, and the guidance trajectory is generalized through the DMP model in the trained skill learning module. The virtual fixture guidance force is calculated according to the current real-time position of the robot and the guidance trajectory, and the virtual fixture guidance force is fed back to the main-end tactile device to guide the novice operator to complete the task.

[0074] Example 2

[0075] This embodiment provides a teleoperation robot assistance method based on vision and imitation learning, comprising the following steps:

[0076] Step 1: Calibrate the coordinate system of the master and slave robots and define the initial position X of the master tactile device l0 =[x l0 ,y l0 , z l0 ] and the initial position X of the slave robot arm f0 =[x f0 ,y f0 , z f0 ], by designing a suitable scaling matrix S = diag(S x , S y , S z ), so that the master-slave displacement satisfies X f =X f0 +S·(X l -X l0 );

[0077] Step 2: Map the real-time pose of the master and slave robots and collect the real-time pose of the tactile device:

[0078] X l (t) = [x l (t),y l (t),z l (t)]

[0079] Then calculate the target pose from the end:

[0080]

[0081] The joint angles are solved through inverse kinematics and sent to the slave robot arm;

[0082] Step 3: Model and generalize the expert operation trajectory using dynamic motion primitives (DMP): First, collect data on the expert trajectory. The expert completes N demonstration operations through the tactile device and records the trajectory sequence from the end to the end. Contains position x, speed Acceleration By building a second-order differential equation, x, and The relationship is as follows:

[0083]

[0084] Among them, K l is the stiffness matrix, D l is the damping matrix, τ is the time scaling factor, f(z) is the nonlinear term, and the phase variable z is then defined as:

[0085]

[0086] Among them, α is the attenuation coefficient, ensuring that z gradually changes from 1 to 0, and the nonlinear term f(z) is defined using the Gaussian basis function as follows:

[0087]

[0088] Among them, c i is the basis function center, σ i is the width, w i is the weight coefficient;

[0089] For position x, and The demonstrated trajectory can obtain the objective function that needs to be learned as follows:

[0090]

[0091] By calculating the minimum value of the following formula, the optimal weighting coefficient w can be obtained: i :

[0092] min∑(f target -f(z)) 2

[0093] Step 4: Use Kinect to capture remote target information. The main task is to establish the relationship between the camera calibration and the remote robot, and convert the target coordinates into robot coordinates through the camera. Assume that the position of the remote robot end effector is x f =[x r ,y r ,z r ], the corresponding position in Kinect is x c =[x c ,y c ,z c ]. f and x c The relationship is defined as follows:

[0094]

[0095] In order to reduce the camera calibration error, this embodiment adopts the LSM method for calibration:

[0096]

[0097] Among them, I 4 is the identity matrix. [x ri ,y ri ,z ri ] i＝n , and [x ci ,yci ,z ci ] are the positions of the i-th point in the remote robot coordinate system and the Kinect coordinate system, respectively. Using the LSM algorithm, we can know that:

[0098] X cL *T cL =X rL

[0099] So we can get:

[0100]

[0101] Substituting the above five formulas, we can get the calibration matrix. With the calibration matrix T, we can know the position in the image corresponds to the position in the robot coordinate system:

[0102]

[0103] Step 5: After obtaining the two-dimensional image information of Kinect, extract the object position based on image processing. First, extract the HSV three-channel map and perform binarization on the three HSV three-channels. Then, perform corrosion and expansion operations on the three binary maps. Then, take the maximum connected area of ​​the three binary maps to obtain a rectangular shape. Finally, take the square area node. After obtaining the square area node and adding the object depth information, extract the three-dimensional position [x oc ,y oc ,z oc ], then, the remote robot coordinate system [x or ,y or ,z or ];

[0104] Step 6: Figure 2 As shown in the figure, after obtaining the target position through the camera, the virtual fixture force can be calculated. First, the position X of the slave robot is obtained in real time. f (t) and the expert trajectory X gen (t), and then calculate the point X on the expert trajectory that is closest to the current slave robot posture gen (t closest ) and calculate the distance:

[0105] X error (t) = X gen (t closest )-X f (t)

[0106] Design the virtual fixture guiding force as:

[0107] F guidance =K vf1 Xerror (t)+K vf2 (X vr -X f (t))

[0108] Among them, K vf1 is the expert trajectory tracking gain, K vf2 The attraction gain of the task target point is then set according to the force selector designed by the present invention to set the magnitude and direction of the virtual fixture force fed back by the master-end tactile device:

[0109]

[0110] The guiding force is activated when the error exceeds a threshold ∈, otherwise free operation is allowed.

[0111] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A teleoperation robot assistance system based on vision and imitation learning, characterized in that: include: Teleoperation module, skill learning module, visual processing module and virtual force assistance module; The teleoperation module realizes remote operation of the robot through an external control device, establishes a two-way communication link with the slave robot, pairs the control space of the external control device with the workspace of the robot through the master-slave space matching mechanism, and adjusts the control response of the robot by adjusting the proportional factor; The skill learning module is used to extract and learn the features of the expert trajectory, build an expert skill library based on the improved dynamic motion primitive algorithm, and generate the best reference trajectory, combined with the real-time status of the robot, to provide guidance for the teleoperation module; The visual processing module generates a depth map through a stereo vision algorithm, establishes a transformation matrix from the camera coordinate system to the robot base coordinate system, performs target recognition through HSV color space conversion and morphological processing, calculates the three-dimensional coordinates of the target object in the camera coordinate system based on the depth map, and converts it into a posture in the robot coordinate system through a pre-calibrated transformation matrix; The virtual force auxiliary module obtains the robot state and the optimal reference trajectory, searches for the optimal target point of the robot on the optimal reference trajectory, constructs the gravitational force of the optimal target point and the task completion point on the end of the robot, and feeds back to the teleoperation module.

2. A teleoperation robot assisted method based on vision and imitation learning, characterized in that: A teleoperated robot assistance system based on vision and imitation learning as claimed in claim 1 is provided, comprising the following steps: Based on the teleoperation module, the control space of the external control device is paired with the workspace of the robot, and the coordinate systems of the master and slave robots are calibrated; Based on the visual processing module, the real-time posture mapping of the master and slave robots is performed, the real-time posture of the tactile device is collected, the target posture of the slave end is calculated, and the joint angle is solved through inverse kinematics and sent to the slave end robot arm; Model and generalize the expert operation trajectory based on the skill learning module, build the expert skill library based on the improved dynamic motion primitive algorithm, and generate the best reference trajectory; Capture remote target information based on Kinect; Extracting the location of objects based on image processing; Obtain the slave robot posture and expert trajectory, calculate the point on the expert trajectory that is closest to the current slave robot posture and calculate the distance; The virtual fixture guiding force is calculated based on the virtual force auxiliary module.

3. The teleoperation robot assisted method based on vision and imitation learning according to claim 2, characterized in that: Based on the teleoperation module, the control space of the external control device is paired with the workspace of the robot, and the coordinate system of the master and slave robots is calibrated, including: Define the initial pose X of the master tactile device l0 =[x l0 ,y l0 , z l0 ] and the initial position X of the slave robot arm f0 =[x f0 ,y f0 , z f0 ], based on the preset scaling matrix S = diag(S x , S y , S z ), so that the master-slave displacement satisfies X f =X f0 +S·(X l -X l0 ).

4. The teleoperation robot assisted method based on vision and imitation learning according to claim 3, characterized in that: Based on the visual processing module, the real-time posture mapping of the master and slave robots is performed, and the real-time posture of the tactile device is collected. The specific expression is as follows: X l (t)=[x l (t),y l (t),z l (t)] Calculate the target pose from the end:

5. The teleoperation robot assisted method based on vision and imitation learning according to claim 2, characterized in that: Model and generalize the expert operation trajectory based on the skill learning module, including: Obtain the expert operation trajectory and record the end-to-end trajectory sequence Contains position x, speed Acceleration By building a second-order differential equation, x, and The relationship is as follows: Among them, K l is the stiffness matrix, D l is the damping matrix, τ is the time scaling factor, f(z) is the nonlinear term, and the phase variable z is then defined as: Among them, α is the attenuation coefficient; The nonlinear term f(z) is defined using the Gaussian basis function as follows: Among them, c i is the basis function center, σ i is the width, w i is the weight coefficient; For position x, and The demonstrated trajectory gives the objective function:

6. The teleoperation robot assisted method based on vision and imitation learning according to claim 2, characterized in that: Capture remote target information based on Kinect, including: Set the position of the remote robot end effector to x f =[x r ,y r ,z r ], the corresponding position in Kinect is x c =[x c ,y c ,z c ]; x f and x c The relationship is defined as follows: Using LSM algorithm for calibration: Where I4 is the identity matrix, [x ri ,y ri ,z ri ] i＝n , and [x ci ,y ci ,z ci ] are the positions of the i-th point in the remote robot coordinate system and the Kinect coordinate system, respectively. Based on the calibration matrix T, the position in the image corresponds to the position of the robot coordinate system:

7. The teleoperation robot assisted method based on vision and imitation learning according to claim 2, characterized in that: Extracting the location of an object based on image processing includes: Get the two-dimensional image information of Kinect, extract the HSV three-channel image and perform binarization processing, perform corrosion and expansion operations on the binary image, take the maximum connected area of ​​the three binary maps, get the rectangular shape, and get the square area nodes, add the object depth information to the square area nodes, extract the three-dimensional position in the camera coordinate system, and calculate the position in the remote robot coordinate system.

8. The teleoperation robot assisted method based on vision and imitation learning according to claim 2, characterized in that: The virtual fixture guiding force is calculated based on the virtual force auxiliary module, which is specifically expressed as: F guidance =K vf1 X error (t)+K vf2 (X vr -X f (t)) X error (t)=X gen (t closest )-X f (t) Among them, F guidance represents the virtual fixture guiding force, X f (t) represents the position of the slave robot, X gen (t closest ) represents the point on the expert trajectory that is closest to the current slave robot posture, K vf1 is the expert trajectory tracking gain, K vf2 Attracts buffs to mission objectives.

9. The teleoperation robot assisted method based on vision and imitation learning according to claim 8, characterized in that: The virtual fixture force fed back by the master tactile device is set according to the force selector, which is specifically expressed as: Among them, X error It represents the distance of the point on the expert trajectory that is closest to the current slave robot posture, and ∈ represents the set threshold.

Images