A robotic arm teleoperation interaction method based on digital twin of target object

Abstract

The present invention relates to a robot arm teleoperation interaction method based on a digital twin of a target object, which generates a digital twin of a real object at a corresponding position of a virtual scene using a prefabricated three-dimensional model through object recognition and positioning, and can put a new prefabricated three-dimensional model into a model library for reconstructing the scene, while integrating the operation of the robot arm into the scene; the user operates the robot arm by directly interacting with the digital twin of the object to be grasped. The robot arm teleoperation interaction method based on a digital twin of a target object of the present invention is a direct interactive operation mode for the target object, which reduces unnecessary collisions caused by user operation errors, and reduces the workload and difficulty of use; the use of prefabricated three-dimensional models instead of point cloud models greatly reduces the amount of data that needs to be transmitted, ensures real-time performance, and improves user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm control, and in particular to a robot arm teleoperation interaction method based on a digital twin of a target object. Background Art

[0002] In traditional systems, users control the remote robot using joysticks, gamepads, keyboards, and mice while receiving visual feedback from a 2D display. Using a 2D display to observe the work scene can cause users to have difficulty in obtaining information about the work scene. A single 2D display can only observe the workspace from one direction. In this case, the user cannot directly perceive the depth and has difficulty judging the distance between the robot end effector and the workpiece. In addition, the user needs to pay attention to the display and controller at the same time during operation, resulting in the user being unable to focus on controlling the robot during the interaction. Although the multi-camera layout allows users to observe and judge the distance between objects from different perspectives, thereby compensating for the limitation of only one monocular 2D camera, the use of multiple 2D camera feedback to avoid collisions between the robot and surrounding objects makes the remote control robot system more complicated, which inevitably leads to inefficient robot operation and increased operator workload.

[0003] Mixed reality (MR) provides a three-dimensional, stereoscopic, virtual and real scene and a more natural and intuitive interaction, making it easier for users to understand the working environment and providing a better user experience. Through MR, 2D visualization is improved to 3D visualization, which can provide a more three-dimensional working scene and a certain depth information. This is mainly to replace the 2D display with a point cloud interface, a pure virtual interface, or an interface that displays the point cloud and the real robot at the same time. The point cloud interface faces the problem of insufficient real-time performance due to excessive data volume; in MR, additional information about the operation of the robot arm can also be displayed to help users better understand the working scene, such as the end position of the robot end device, the safety area of ​​the robot arm, and the planned robot arm motion trajectory; MR can also be used to change the operation method of the robot arm, mainly using voice recognition and head gaze to select the posture of the virtual gripper, moving the virtual end device through gestures to control the real robot arm, and using a series of waypoints (waypoints are the positions to which the robot end device must move) to create and edit the robot's movement.

[0004] Although the MR interface brings a better user experience to the remote operation of the robotic arm, the current three-dimensional virtual scene is mainly presented in the form of point cloud, and the amount of data in the point cloud model is often very large, which makes real-time scene modeling, data conversion and description very difficult. At the same time, the point cloud model can only display the area facing the scanning device, and there is no point cloud display for the back of the object that cannot be scanned. In terms of operation, the current remote operation method of the robotic arm based on mixed reality is mainly to operate the end effector. This method is not intuitive enough in the MR environment and the operation is complicated, and it is prone to collision in a multi-object environment. Summary of the invention

[0005] The present invention aims to solve the problems of excessive reconstruction process data volume, lack of intuitiveness in MR environment and complex operation in the existing mixed reality-based robotic arm teleoperation method, and provide a robotic arm teleoperation interaction method based on the digital twin of the target object. The interaction method of the present invention is a direct interactive operation method for the target object, which reduces unnecessary collisions caused by user operation errors, and reduces the workload and difficulty of use; using prefabricated three-dimensional models instead of point cloud models greatly reduces the amount of data that needs to be transmitted, ensures real-time performance, and improves user experience.

[0006] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0007] A robotic arm teleoperation interaction method based on a target object digital twin comprises the following steps:

[0008] The system applicable to the robotic arm teleoperation interaction method includes: a robotic arm, a server, an RGB camera, a depth camera and a HoloLens device; the depth camera, the RGB camera, the robotic arm and the HoloLens device are respectively connected to the server, and the depth camera and the RGB camera are fixed on the robotic arm;

[0009] Step 1: Obtain RGB images and depth images of objects around the robot arm through the RGB camera and depth camera. The server reads the obtained RGB images and depth images, and uses the obtained images to identify and locate objects in the scene;

[0010] Step 2: After obtaining the category and location of the object in step 1, the server sends the relevant information to the HoloLens device, and the HoloLens device generates a digital twin of the real object at the corresponding location in the virtual scene;

[0011] Step 3: The user uses the gesture recognition of the HoloLens device to grab the virtual object and move it to the desired location. After the user presses the virtual confirmation button, the HoloLens device detects all virtual objects, determines the moved virtual objects, and sends the categories and target locations of these virtual objects to the server. The server autonomously controls the robotic arm to move the corresponding real object from the current location to the target location.

[0012] In the above technical solution, the steps of the server using the acquired image to identify the object in the scene in step 1 are as follows:

[0013] To identify different objects by color, the server performs Gaussian blur processing, HSV image conversion, and corrosion processing on the RGB image in sequence, and then sets the color range of the HSV image to obtain the bounding box of the corresponding color and represents it with four values ​​(x, y, w, h), where (x, y) represents the upper left corner of the bounding box, and (w, h) represents the width and height of the bounding box. The pixel coordinates of the center of the object's bounding box in the RGB image are calculated in this way.

[0014] In the above technical solution, the steps of locating the object in the scene by the server using the acquired image in step 1 are as follows:

[0015] According to the pixel coordinates of the center of the object's bounding box in the RGB image, determine its corresponding position in the depth image. The value corresponding to this position is the required depth value of the object. Multiply the pixel coordinates of the object in the depth image by the intrinsic parameter matrix and depth value of the depth camera to obtain the coordinates of the object in the depth camera coordinates. Multiply the coordinates of the object in the depth camera coordinates by the coordinate transformation matrix from the depth camera coordinate system to the robotic arm base coordinate system to obtain the coordinates of the object in the robotic arm base coordinate system.

[0016] In the above technical solution, the relevant information in step 2 includes: the category of the object, the three-dimensional coordinates representing the position of the object, and the quaternion representing the posture of the object.

[0017] In the above technical solution, in step 2, after the HoloLens device receives the relevant information of the object, it will use the virtual robotic arm base coordinate system as the reference coordinate system to generate a digital twin of the real object at the corresponding position of the virtual scene.

[0018] In the above technical solution, the step of generating a digital twin of a real object at a corresponding position of a virtual scene in step 2 is specifically as follows:

[0019] Use Unity software to generate a virtual scene in the HoloLens device, create a class ObjMsg in Unity to store relevant information of the object, including the object's name, 3D coordinates posx, posy, posz and quaternions rotx, roty, rotz, rotw, create a ROS information receiving script, and use Subscriber() to receive ROS information. After Unity receives the object information from ROS, it stores the information in the hash table ObjectPoseNow using the ObjectPoseNow.Add() function. The hash table uses the object name as the key and the 3D coordinates and quaternion as the value. The ObjAdd() function is used in the order of reception to add the prefabricated 3D model of the object to the virtual scene and generate a digital twin of the real object.

[0020] In the above technical solution, the depth camera and the server, the RBG camera and the server, and the robotic arm and the server are all connected by wire, and the server and the HoloLens device are connected by wireless.

[0021] In the above technical solution, the server and the HoloLens device communicate with each other using ROS-TCP-Connector and ROS-TCP-Endpoint.

[0022] The beneficial effects of the present invention are:

[0023] The robot arm remote operation interaction method based on the digital twin of the target object of the present invention generates a digital twin of the real object at the corresponding position of the virtual scene using a prefabricated three-dimensional model through object recognition and positioning, and the new prefabricated three-dimensional model can be placed in the model library for reconstructing the scene, and the operation of the robot arm is combined with the scene. Compared with traditional 2D image data, three-dimensional data provides depth information of the object, which improves the efficiency of the user's remote operation and the success rate of completing work tasks. In order to allow users to not pay attention to the display and controller at the same time, the present invention combines the three-dimensional scene and the operation of the robot arm together, allowing users to operate the robot arm while observing the virtual environment. Compared with the reconstruction of the three-dimensional point cloud scene, the present invention uses a prefabricated three-dimensional model instead of a point cloud model, which greatly reduces the amount of data that needs to be transmitted, ensures real-time performance, and improves the user experience. At the same time, the virtual scene reconstructed by the present invention can be observed from any angle to obtain the relative relationship between the object and the robot arm.

[0024] In the robot arm teleoperation interaction method based on the digital twin of the target object of the present invention, the user controls the robot arm by directly interacting with the digital twin of the target object, rather than the user operating the robot arm through physical devices such as rockers and handles or moving the end effector of the virtual robot arm in a virtual scene to control the robot arm. The user of the present invention directly interacts with the digital twin of the target object, which can reduce the operator's operating difficulty and learning difficulty, and at the same time reduce unnecessary collisions caused by operator errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0026] Figure 1 This is a flow chart of the robotic arm teleoperation interaction method based on the digital twin of the target object of the present invention.

[0027] Figure 2 Schematic diagram of the interactive interface, where (a) is the initial interface and (b) is the interface for adding virtual objects.

[0028] Figure 3 Schematic diagram of information exchange.

[0029] Figure 4 The figure is a flow chart of the robotic arm teleoperation interaction method based on the digital twin of the target object. DETAILED DESCRIPTION

[0030] The present invention is described in detail below with reference to the accompanying drawings.

[0031] Combination Figure 1 , specifically describing a robot arm teleoperation interaction method based on a target object digital twin of the present invention, comprising the following steps:

[0032] The system applicable to the robotic arm teleoperation interaction method includes: a robotic arm, a server, an RGB camera, a depth camera and a HoloLens device; the RGB camera and the server, the depth camera and the server, and the robotic arm and the server are all connected by wire, and the server and the HoloLens device are connected wirelessly. Specifically: the server and the HoloLens device communicate using ROS-TCP-Connector and ROS-TCP-Endpoint; the RGB camera and the depth camera are fixed on the robotic arm; Figure 3 Demonstrate information exchange between components;

[0033] Step 1: Obtain RGB images and depth images of objects around the robot arm through the RGB camera and depth camera. The server reads the obtained RGB images and depth images, and uses the obtained images to identify and locate objects in the scene;

[0034] The server uses the acquired image to identify the objects in the scene in the following steps:

[0035] To identify different objects by color, the server performs Gaussian blur processing, HSV image conversion, and corrosion processing on the RGB image in sequence, and then sets the color range of the HSV image to obtain the bounding box of the corresponding color and represents it with four values ​​(x, y, w, h), where (x, y) represents the upper left corner of the bounding box, and (w, h) represents the width and height of the bounding box. The pixel coordinates of the center of the object's bounding box in the RGB image are calculated in this way.

[0036] The steps for the server to locate objects in the scene using the acquired images are as follows:

[0037] According to the pixel coordinates of the center of the object bounding box in the RGB image, determine its corresponding position in the depth image. The value corresponding to this position is the required depth value of the object. Multiply the pixel coordinates of the object in the depth image by the intrinsic parameter matrix and depth value of the depth camera to obtain the coordinates of the object in the depth camera coordinates. Multiply the coordinates of the object in the depth camera coordinates by the coordinate transformation matrix from the depth camera coordinate system to the robot base coordinate system to obtain the coordinates of the object in the robot base coordinate system.

[0038] Step 2: After obtaining the object category and location in step 1, the server sends the relevant information (including: object category, 3D coordinates representing the object location, quaternion representing the object posture) to the HoloLens device, and the HoloLens device generates a digital twin of the real object at the corresponding position in the virtual scene; see the specific scene for details. Figure 2 ;

[0039] In the above steps, after the HoloLens device receives the relevant information of the object, it uses the virtual robotic arm base coordinate system as the reference coordinate system to generate a digital twin of the real object at the corresponding position of the virtual scene, specifically:

[0040] Use Unity software to generate a virtual scene in the HoloLens device. Create a class ObjMsg in Unity to store relevant information of the object, including the object's name, 3D coordinates posx, posy, posz, and quaternions rotx, roty, rotz, and rotw. Create a ROS information receiving script in which Subscriber() is used to receive ROS object information. After Unity receives the ROS object information, it uses the ObjectPoseNow.Add() function to store the information in the hash table ObjectPoseNow. The hash table uses the object name as the key and the 3D coordinates and quaternion as the value. The ObjAdd() function is used in the order of reception to add the prefabricated 3D model of the object to the virtual scene and generate a digital twin of the real object.

[0041] Step 3: The user uses the gesture recognition of the HoloLens device to grab the virtual object and move it to the desired location. After the user presses the virtual confirmation button, the HoloLens device detects all virtual objects, determines the moved virtual objects, and sends the categories and target locations of these virtual objects to the server. The server autonomously controls the robotic arm to move the corresponding real object from the current location to the target location.

[0042] The following is a more detailed description of the robotic arm teleoperation interaction method based on the digital twin of the target object of the present invention in the form of a software module.

[0043] The proposed method of remote operation interaction of a robotic arm based on digital twin of a target object is based on the design and development of three main software modules. The first module is responsible for the identification and positioning of graspable objects in the working scene of the robotic arm. The second module is responsible for receiving object information and generating a virtual scene for user operation. The third module is responsible for controlling the robotic arm to autonomously complete the grasping after receiving the task target. The overall process of the method of remote operation interaction of a robotic arm based on digital twin of a target object of the present invention is as follows: Figure 1As shown. When the HoloLens device, the robotic arm, and the server are all turned on and the HoloLens device is successfully connected to the server, the RGB camera is successfully connected to the server, the depth camera is successfully connected to the server, and the server is successfully connected to the robotic arm. First, the RGB image and depth image of the objects in the surrounding environment of the robotic arm are obtained through the RGB camera and the depth camera, and the objects in the scene are identified and located using the obtained images. Secondly, after obtaining the category and position of the object, the relevant information is sent to the HoloLens device, and the HoloLens device generates a digital twin of the real object at the corresponding position of the virtual scene. Finally, the user grabs the virtual object through the gesture recognition of the HoloLens device and moves it to the desired position. After confirming that it is correct, click the confirmation button, and the HoloLens device will detect the virtual objects that have been moved, and send the category and current position of these objects to the server. The server autonomously controls the robotic arm to move the corresponding real object from the current position to the target position.

[0044] In the first module of the recognition and location of graspable objects, we identify different objects by color. The server performs Gaussian blur processing, HSV image conversion, and erosion processing on the RGB image in sequence. By setting the color range of the HSV image, the bounding box of the corresponding color is obtained and represented by four values ​​(x, y, w, h), where (x, y) represents the upper left corner of the bounding box, and (w, h) represents the width and height of the bounding box. The pixel coordinates of the center of the object's bounding box in the RGB image are calculated. The positioning operation is mainly to determine the corresponding position in the depth map based on the pixel coordinates of the center of the object's bounding box in the RGB image. The value corresponding to this position is the required depth value. Since the resolutions of the depth image and the RGB image are different, they also need to be aligned, but there is no need to align the entire image. It is only necessary to convert the pixel coordinates of several center points. After obtaining the image coordinates of the object and its depth value, multiply them by the coordinate transformation matrix from the depth camera coordinate system to the robot base coordinate system to obtain the coordinates of the object in the robot base coordinate system. Since the depth camera and the RGB camera are fixed on the robot in the present invention, their positions are relatively fixed, and their transformation relationship can be directly added to a coordinate transformation manager in the ROS TF tree. Through the ROS TF tree, the conversion between the depth camera optical coordinate system, the end effector coordinate system, and the robot base coordinate system can be realized in real time.

[0045] The second module is to receive object information and generate a virtual scene for users to operate. This requires communication between the server and the HoloLens device. We use ROS-TCP-Connector and ROS-TCP-Endpoint to achieve this function. On the HoloLens device side, through ROS-TCP-Connector, the HoloLens device can subscribe to the topics published by ROS in the server and obtain object-related information, including the object's category, location, and posture.

[0046] After the HoloLens device receives the information of the object, it will use the base of the virtual robotic arm as the reference coordinate system to generate a digital twin of the real object at the corresponding position. The digital twin of these objects can be moved at will by the user. Figure 2 As shown. Virtual objects have physical properties and can simulate the movement, collision, and falling of objects in real scenes. At the same time, users can observe virtual scenes from any angle to confirm the relative positions between objects. When users observe virtual scenes, they can also observe the surrounding real environment, including the workplace of the robot arm, which allows users to promptly discover accidents during the movement of the robot arm and press the emergency stop button to prevent greater danger. Specifically, we created a class ObjMsg to store relevant information about objects, including the name of the object, three-dimensional coordinates posx, posy, posz, and quaternions rotx, roty, rotz, and rotw. In the ROS information receiving script, Subscriber() is used to receive ROS information. After unity receives the object information from ROS, it uses the ObjectPoseNow.Add() function to store its information in the hash table ObjectPoseNow. The hash table uses the object name as the key, the three-dimensional coordinates and the quaternion as the value, and uses the ObjAdd() function in the order of reception to add the prefabricated three-dimensional model of the object to the virtual scene and generate a digital twin of the real object. The core algorithm is as follows:

[0047]

[0048] In the third module, the user needs to move virtual objects in the virtual scene of the HoloLens device to indicate the target object and end point of the task. The user can move objects to any position in the virtual scene without affecting the real scene until the user confirms that all virtual objects have moved to the target position and then presses the confirmation button. After the user confirms, the HoloLens device will scan all objects in the virtual scene and calculate the difference between the current position and the recorded position of each object. If the difference is less than the set threshold, it is considered that the object has not moved. Otherwise, the current coordinates of the object are sent to ROS. Specifically, we wrote a script sent by ROS. The script traverses all the object information obj in the virtual scene. now , with the name of the current object obj name For input, use the ObjectPoseNow() function to obtain the object information obj stored in the ROS receiving script saved , and judge whether the current virtual object is moved according to formula (1). As long as one of the above conditions is met, the current virtual object can be regarded as moved:

[0049] |x now -x saved |>10 -5

[0050] |y now -y saved |>10 -5

[0051] |z now -z saved |>10 -5 (1)

[0052] where x now ,y now , z now Object information obj in Unity now The current three-dimensional coordinates of x saved ,y saved , z saved Represents the object information obj stored in the script saved In the virtual scene, since the data type of the object coordinates is float, its precision is about 6-9 digits, so we set the threshold to 10. -5 , to ensure that the non-moving object will not be misjudged as a moving object due to data errors, causing the robot arm to execute unnecessary commands. After searching for the virtual object to be moved, update its current 3D coordinates to the corresponding obj saved Then the updated object information obj saved Send to ROS through the Publish() function. The core algorithm is shown in the following table:

[0053]

[0054] The program sends the task object category and end position information to ROS through ROS-TCP-Connector in the form of a topic, and ROS subscribes to this topic through ROS-TCP-Endpoint. Then ROS reads the current position of the target object, determines the coordinates to be reached by the end effector of the robot arm, uses the inverse kinematics solver to obtain the position and posture of the robot arm, and then controls the robot arm to move autonomously to the current position of the target object and grab the object. Similarly, ROS obtains the position and posture of the robot arm at the end position through the inverse kinematics solver, and autonomously controls the robot arm to go there and place the object at the end position. Finally, the robot arm is controlled to return to the initial position without collision and wait for the next task.

[0055] Figure 3 The information exchange between the devices in the system to which the method of the present invention is applicable is demonstrated, and the devices included are a depth camera, an RGB camera, a server, a robotic arm, and a HoloLens device. In this system, the RGB camera and the server, the depth camera and the server, and the robotic arm and the server are all connected by wire, and the server and the HoloLens are connected wirelessly. In addition, the depth camera and the RGB camera are fixed on the robotic arm. The RGB camera and the depth camera need to transmit the RGB image and the depth image of the scene to the server for object recognition and positioning. The server sends the object category, three-dimensional coordinates, and the quaternion representing the object posture to the HoloLens device to generate a virtual scene. The user indicates the task content to the server through the virtual interface, and the server uses the control command of the robotic arm to autonomously complete the grasping and placement task according to the task content.

[0056] The specific operation method of the interactive mode involved in the method of the present invention is as follows:

[0057] 1. The user wears the HoloLens device, starts the interactive program, starts the control program of the robotic arm and the server, and waits for the virtual mapping of the real scene to be displayed on the HoloLens device.

[0058] After the user starts the interactive program, a desktop with a virtual robot arm and a button will appear in front of him. Figure 2 As shown in (a), the robot arm is used to help the user locate the object to be grasped, and the button is used to confirm the task to prevent user operation errors. The HoloLens device displays the prefabricated 3D model corresponding to the real object at the corresponding position of the virtual scene according to the object information sent by the server for user operation. Figure 2 (b) as shown.

[0059] 2. Users can use the HoloLens device to observe the virtual mapping of objects in the real scene. If they want to operate, they pinch the virtual object with two fingers and drag it to the desired position, and then click the button to confirm the task.

[0060] 3. The server obtains the object to be moved and the target position through the HoloLens device, and autonomously controls the robotic arm to move the object based on this information.

[0061] After the user confirms the task content, there is no need for any operation, waiting for the server to control the robotic arm to complete autonomous grasping and placement. During this period, the user can directly see the real working scene of the robotic arm, discover accidents in time, and stop the robotic arm in an emergency. After waiting for the robotic arm to complete the current task, the user can continue to instruct the task on the HoloLens device.

[0062] In summary, the robot arm remote operation interaction method based on the digital twin of the target object of the present invention generates the digital twin of the real object at the corresponding position of the virtual scene using the prefabricated three-dimensional model through object recognition and positioning, and the new prefabricated three-dimensional model can be placed in the model library for reconstructing the scene, and the operation of the robot arm is integrated into the scene; the user operates the robot arm by directly interacting with the digital twin of the object to be grasped, rather than moving the end effector of the virtual robot arm. Therefore, the method of the present invention is a direct interactive operation mode for the target object, which reduces unnecessary collisions caused by user operation errors, and reduces the workload and difficulty of use; the use of prefabricated three-dimensional models instead of point cloud models greatly reduces the amount of data that needs to be transmitted, ensures real-time performance, and improves user experience.

[0063] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A robotic arm teleoperation interaction method based on a target object digital twin, characterized in that: The following steps are involved: The system applicable to the robotic arm teleoperation interaction method includes: a robotic arm, a server, an RGB camera, a depth camera and a HoloLens device; the depth camera, the RGB camera, the robotic arm and the HoloLens device are respectively connected to the server, and the depth camera and the RGB camera are fixed on the robotic arm; Step 1: Obtain RGB images and depth images of objects around the robot arm through the RGB camera and depth camera. The server reads the obtained RGB images and depth images, and uses the obtained images to identify and locate objects in the scene; Step 2: After obtaining the category and location of the object in step 1, the server sends the relevant information to the HoloLens device, and the HoloLens device generates a digital twin of the real object at the corresponding location in the virtual scene; The relevant information in step 2 includes: the category of the object, the three-dimensional coordinates representing the position of the object, and the quaternion representing the posture of the object; In step 2, after the HoloLens device receives the relevant information of the object, it uses the virtual robotic arm base coordinate system as the reference coordinate system to generate a digital twin of the real object at the corresponding position of the virtual scene; The steps to generate a digital twin of a real object at the corresponding position in the virtual scene are as follows: Use Unity software to generate a virtual scene in the HoloLens device. Create a class ObjMsg in Unity to store relevant information of the object, including the object's name, 3D coordinates posx, posy, posz, and quaternions rotx, roty, rotz, and rotw. Create a ROS information receiving script in which Subscriber() is used to receive ROS object information. After Unity receives the ROS object information, it uses the ObjectPoseNow.Add() function to store the information in the hash table ObjectPoseNow. The hash table uses the object name as the key and the 3D coordinates and quaternion as the value. The ObjAdd() function is used in the order of reception to add the prefabricated 3D model of the object to the virtual scene and generate a digital twin of the real object. Step 3: The user uses the gesture recognition of the HoloLens device to grab the virtual object and move it to the desired location. After the user presses the virtual confirmation button, the HoloLens device detects all virtual objects, determines the moved virtual objects, and sends the categories and target locations of these virtual objects to the server. The server autonomously controls the robotic arm to move the corresponding real object from the current location to the target location.

2. The robot arm teleoperation interaction method according to claim 1, characterized in that: In step 1, the server uses the acquired image to identify the objects in the scene as follows: To identify different objects by color, the server performs Gaussian blur processing, HSV image conversion, and corrosion processing on the RGB image in sequence, and then sets the color range of the HSV image to obtain the bounding box of the corresponding color and represents it with four values ​​(x, y, w, h), where (x, y) represents the upper left corner of the bounding box, and (w, h) represents the width and height of the bounding box. The pixel coordinates of the center of the object's bounding box in the RGB image are calculated in this way.

3. The robot arm teleoperation interaction method according to claim 2, characterized in that: In step 1, the server uses the acquired image to locate the object in the scene as follows: According to the pixel coordinates of the center of the object's bounding box in the RGB image, determine its corresponding position in the depth image. The value corresponding to this position is the required depth value of the object. Multiply the pixel coordinates of the object in the depth image by the intrinsic parameter matrix and depth value of the depth camera to obtain the coordinates of the object in the depth camera coordinates. Multiply the coordinates of the object in the depth camera coordinates by the coordinate transformation matrix from the depth camera coordinate system to the robotic arm base coordinate system to obtain the coordinates of the object in the robotic arm base coordinate system.

4. The robot arm teleoperation interaction method according to claim 1, characterized in that: The depth camera and the server, the RBG camera and the server, and the robotic arm and the server are all connected by wire, and the server and the HoloLens device are connected wirelessly.

5. The robot arm teleoperation interaction method according to claim 1, characterized in that: The server and the HoloLens device communicate using ROS-TCP-Connector and ROS-TCP-Endpoint.

Images