Mechanical arm teaching system based on VR and control method thereof

Abstract

The invention relates to the technical field of industrial automation and intelligent manufacturing, and provides a VR-based mechanical arm teaching system and a control method thereof.The VR-based mechanical arm teaching system comprises VR base stations which are evenly arranged around a working area of a mechanical arm in a distributed mode and irradiate a VR positioner by emitting lasers at different angles to the working space; the VR positioner receives the laser at different angles, and reversely positions the pose of the teaching pen in the coordinate system of the VR equipment; and the controller reads the pose of the teaching pen under the coordinate system of the VR equipment, calls the hand-eye calibration result, performs coordinate transformation on the pose under the coordinate system of the VR equipment to obtain the pose of the teaching pen under the coordinate system of the mechanical arm base, and takes the pose of the teaching pen under the coordinate system of the mechanical arm base as mapping to obtain a teaching track control target. And a control instruction is edited and issued to the mechanical arm communication protocol, and mechanical arm teaching control is achieved. And the mechanical arm can smoothly reproduce the track with high precision and is adaptive to complex working conditions.

Description

Technical Field

[0001] This invention relates to the field of industrial automation and intelligent manufacturing technology, and in particular to a VR-based robotic arm teaching system and its control method. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In the field of industrial automation and intelligent manufacturing, robotic arms, as core execution units, directly determine the flexibility and intelligence level of production lines through the efficiency and accuracy of their trajectory planning and motion teaching. In complex trajectory operations such as welding and gluing, the efficiency and accuracy of teaching the robotic arm's motion path directly determine the production line's response speed and product quality. Traditional teaching programming heavily relies on operators manually recording the position and orientation of the robotic arm's end effector TCP (tool center point) point by point using a handheld teach pendant. This method is not only cumbersome and time-consuming, but more importantly, when faced with complex three-dimensional curves, operators struggle to accurately and intuitively define paths using abstract joint coordinates or a world coordinate system. This "what you see is not what you get" programming model has become a major obstacle to improving production flexibility and rapid deployment capabilities.

[0004] To enhance intuitiveness, some technical solutions attempt to use visual capture or simple pointing devices for teaching, but these solutions have inherent flaws in accuracy and reliability. For example, motion capture systems based on ordinary cameras are susceptible to ambient light interference and heavily rely on visual algorithms to identify feature points, with limited positioning accuracy, making it difficult to meet the trajectory reproduction requirements of industrial-grade welding and gluing. More critically, these solutions generally lack a high-precision core mechanism for uniformly calibrating the coordinate system of the teaching device with the robotic arm's TCP coordinate system, resulting in uncalibrated systematic errors between virtual commands and real actions.

[0005] At the software level, while some robot simulation and offline programming platforms exist, they are typically closed, high-cost commercial systems with fixed functions and limited customization to meet specific process requirements. Furthermore, these platforms generally lack effective support for cross-platform deployment (such as Windows, Linux, and Android), restricting their application in diverse industrial scenarios (such as integration with robots from different brands or monitoring on mobile terminals). Their built-in UI interfaces are mostly designed for mouse and keyboard use, failing to provide data visualization and control functions that align with the immersive interaction logic of VR environments.

[0006] In summary, existing robotic arm teaching technologies suffer from a core contradiction: easy-to-use solutions (such as traditional teach pendants and simple vision systems) are insufficient in terms of accuracy and efficiency for complex paths; while solutions that pursue precision (such as high-end offline programming) suffer from problems such as closed systems, unintuitive interaction, and difficulty in calibrating with the environment. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a VR-based robotic arm teaching system and its control method. Multi-base station collaborative positioning reduces occlusion interference, and by combining sub-millimeter-level VR laser positioning with professional calibration algorithms, trajectory data is accurately mapped, avoiding the accuracy errors and trajectory unevenness problems of traditional teaching. This ensures that the robotic arm can reproduce the trajectory with high precision and smoothness, adapting to complex working conditions.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a VR-based robotic arm teaching system.

[0009] The VR-based robotic arm teaching system includes a teaching pen, a robotic arm, a controller, and several VR base stations, with a VR locator installed on the teaching pen. The VR base stations are distributed and evenly deployed around the working area of ​​the robotic arm, and the VR positioners are illuminated by lasers emitted at different angles into the working space. The VR locator is used to receive lasers from different angles and to reverse locate the position of the teaching pen in the VR device coordinate system. The controller is used to read the pose of the teaching pen in the VR device coordinate system, call the hand-eye calibration results, transform the pose in the VR device coordinate system to obtain the pose of the teaching pen in the robot arm base coordinate system, use the pose of the teaching pen in the robot arm base coordinate system as a mapping to obtain the teaching trajectory control target, and edit and publish control commands to the robot arm communication protocol to realize robot arm teaching control.

[0010] Furthermore, the logic output circuit of the teaching pen is connected to several metal spring pins. When the pins of different groups of metal spring pins are shorted, different event signals are sent to the controller.

[0011] Furthermore, the distance between the VR base station and the robotic arm is: , among which, (V x V y V z ) represents the workspace dimensions of the robotic arm, θ represents the field of view of the VR base station, and R arm This represents the maximum turning radius of the robotic arm.

[0012] Furthermore, the distance between the VR base station and the robotic arm satisfies D opt ≤L maxIf / k is not satisfied, then replace it with a larger L. max VR base stations, of which L max is the maximum effective positioning distance of the VR base station, and k is the redundancy coverage coefficient.

[0013] Furthermore, the optimal number of VR base stations is: Where θ is the field of view of the VR base station. For single base station positioning accuracy, This is the positioning error threshold.

[0014] Furthermore, the VR locator is connected to the end joint of the robotic arm via 3D-printed firmware, and the sensor protrusion side of the VR locator faces away from the TCP end of the robotic arm.

[0015] A second aspect of the present invention provides a control method for a VR-based robotic arm teaching system.

[0016] The control method for a VR-based robotic arm teaching system, applicable to the VR-based robotic arm teaching system as described in the first aspect, includes: Read the pose of the teaching pen in the VR device coordinate system, call the hand-eye calibration results, transform the pose in the VR device coordinate system to obtain the pose of the teaching pen in the robot arm base coordinate system. The teaching trajectory control target is obtained by mapping the pose of the teaching pen in the coordinate system of the robotic arm base, and the control commands are edited and issued to the robotic arm communication protocol to realize the teaching control of the robotic arm.

[0017] Furthermore, during the hand-eye calibration process, the VR positioner installed on the robotic arm moves with the robotic arm TCP in the working area. The pose of the VR positioner installed on the robotic arm in the VR device coordinate system is determined by the position of the VR positioner installed on the robotic arm in the VR device coordinate system. Then, through a rigid transformation from the VR positioner installed on the robotic arm to the robotic arm TCP, the pose of the robotic arm TCP in the VR device coordinate system is obtained. Finally, the pose of the robotic arm TCP in the base coordinate system is obtained through the lower computer of the robotic arm. Thus, the pose of the robotic arm in the VR device coordinate system and the robotic arm base coordinate system at the same absolute position is obtained.

[0018] Furthermore, the rigidity transformation is calculated using a CAD 3D model of the robotic arm with the VR positioner installed.

[0019] Furthermore, hand-eye calibration is implemented using the Tsai algorithm according to the calibration type where the eye is outside the hand.

[0020] Compared with the prior art, the beneficial effects of the present invention are: This invention proposes multi-base station collaborative positioning to reduce occlusion interference. It combines sub-millimeter-level VR laser positioning with professional calibration algorithms to accurately map trajectory data, avoiding the accuracy errors and trajectory unevenness problems of traditional teaching. This ensures that the robotic arm can reproduce the trajectory with high precision and smoothness, and is suitable for complex working conditions.

[0021] This invention transforms robotic arm teaching into an intuitive operation of drawing with a handheld teaching pen, which can be mastered without complicated training, greatly reducing the learning threshold and improving teaching efficiency and flexibility. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 A structural diagram of a VR-based robotic arm teaching system provided in an embodiment of the present invention; Figure 2 A top view of a VR-based robotic arm teaching system provided in an embodiment of the present invention; Figure 3 The following is a schematic diagram of the installation method of the VR positioner for the end joint of the robotic arm and the pose of the VR device coordinate system and TCP coordinate system provided in the embodiments of the present invention; Figure 4 This is a schematic diagram of the external structure of the teaching pen provided in an embodiment of the present invention; Figure 5 This is a disassembled diagram of the pen barrel structure provided in an embodiment of the present invention; Figure 6 The internal circuit diagram of the teaching pen provided in the embodiment of the present invention; Figure 7 A schematic diagram of the control program initialization interface of the teaching control system based on the Unity platform provided in the embodiment of the present invention, which runs on a mobile phone in the form of an APP. Figure 8 This is a schematic diagram of the joystick control UI provided in an embodiment of the present invention; Figure 9 A schematic diagram of a data visualization reading interface provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the UI interface for the hand-eye calibration function provided in an embodiment of the present invention; Figure 11 The control flowchart provided for embodiments of the present invention. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] In this invention, terms such as "upper," "lower," "left," "right," "front," "back," "vertical," "horizontal," "side," and "bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only to facilitate the description of the structural relationships of the various components or elements of this invention and do not specifically refer to any component or element in this invention. They should not be construed as limiting the invention.

[0028] In this invention, terms such as "fixed connection," "connected," and "linked" should be interpreted broadly, indicating a fixed connection, an integral connection, or a detachable connection; a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can determine the specific meaning of these terms in this invention based on the specific circumstances, and they should not be construed as limitations on the invention.

[0029] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0030] Example 1 Embodiment 1 of the present invention provides a VR-based robotic arm teaching system.

[0031] The VR-based robotic arm teaching system provided in this embodiment uses a sub-millimeter precision spatial positioning system composed of a VR base station and a locator, along with matching hand-eye coordinate transformation and calibration software, to solve the core problem of difficulty in ensuring coordinate uniformity and calibration accuracy in teaching systems.

[0032] The VR-based robotic arm teaching system provided in this embodiment integrates high-precision spatial positioning, intuitive immersive 3D interaction, reliable coordinate system calibration, and flexible cross-platform software control, achieving fast and high-precision robotic arm teaching with "what you point to is what you get".

[0033] The VR-based robotic arm teaching system provided in this embodiment, such as Figure 1 and Figure 2 As shown, it includes a teaching platform 1, a teaching pen 2 equipped with a VR locator, a robotic arm 6 equipped with a VR locator, a VR positioning base station group, and a controller.

[0034] Robotic arm 6 is fixedly mounted on teaching platform 1, such as Figure 3 As shown, the system includes an end effector 4 for the robotic arm, a VR positioner 5 mounted on the end joint of the robotic arm, and 3D-printed firmware 9 for mounting the VR positioner. The VR positioner 5, mounted on the end joint of the robotic arm, is used to locate the six-DOF pose of the robotic arm's TCP in the VR device's positioning space. This pose serves as a feedback reference for compensating for the difference in TCP pose output from the robotic arm's end-user coordinate system to the base coordinate system, thereby achieving feedback control and improving teaching accuracy. Additionally, during hand-eye calibration, the controller collects the pose of this positioner in the VR positioning system as data input for hand-eye calibration. The results of the hand-eye calibration are used to solve for the transformation matrix from the VR device coordinate system to the robotic arm's base coordinate system.

[0035] like Figure 4 , Figure 5 and Figure 4 As shown, the teaching pen 2 includes a VR positioner 11 mounted on the teaching pen, an internal circuit board 12, and a teaching pen housing 13. The pen tip of the teaching pen 2 is mounted on the VR positioner 11. The teaching pen 2 is connected to the pin port of the VR positioner 11 via the internal circuit board 12. By pressing the various buttons on the teaching pen 2, specific signals are output to the pin port of the VR positioner 11 controlled by the logic circuit to trigger events such as "data acquisition," "planning and execution," and "hand-eye calibration" designed by the control system. In addition to the logic circuit module, the teaching pen 2 also includes a keyboard module, a power supply module, and other circuit modules. By holding the teaching pen 2 and drawing the teaching trajectory 3 in the workspace of the robotic arm, teaching data can be quickly entered into the control system.

[0036] The teaching pen casing 13 is 3D printed and includes three buttons: a "Data Acquisition" button, a "Hand-Eye Calibration" button, and a "Plan and Execute" button, as well as a power button and an LED indicator. The teaching pen tip is equipped with a VR locator, which has a built-in Bluetooth wireless communication module for communication between the teaching pen and the control system. The teaching pen is powered by a power module containing a button battery and includes a keyboard detection module. The core function of the teaching pen is a logic output circuit, which is connected to the metal spring pins on the back of the VR locator, such as... Figure 6As shown, when the "Acquire Data" button is pressed, a specific set of two spring pins is shorted via a logic circuit. When the "Hand-Eye Calibration" button is pressed, another specific set of two spring pins is shorted via the logic circuit. When the "Planning and Execution" button is pressed, yet another specific set of two spring pins is shorted via the logic circuit. When the pins of different sets of metal spring pins on the back of the VR locator are shorted, a specific event signal is sent to the control system. After the control system receives and analyzes the specific event signal, it automatically runs the corresponding control script software. By operating the teaching pen button, the logic circuit triggers the specific pin configuration of the VR locator, enabling the VR locator to send control signals to the controller, thus realizing the control of each functional module via the teaching pen button during the teaching process.

[0037] The VR positioning base station group includes several VR base stations 7, which are distributed and evenly deployed around the working area of ​​the robotic arm 6. The VR base stations continuously emit infrared laser beams to scan the entire space. The VR locator calculates the time difference of arrival of each laser beam and combines the perspective of different VR base stations 7 to accurately calculate the position and orientation of the VR locator in three-dimensional space, so as to realize the reverse positioning of the VR locator. The more VR base stations 7 there are and the more evenly they are deployed in the working space, the higher the positioning accuracy of the VR locator.

[0038] In other words, multiple VR base stations 7 are distributed around the workspace. By emitting lasers at different angles into the workspace to illuminate the VR locator, the VR locator can determine its own pose in the workspace based on the different angles of laser light received by the sensor, thus achieving positioning. The multiple VR base stations automatically calibrate the relative pose of each base station in the workspace through device management software, and achieve collaborative work through automatic calibration and association to locate the VR locator in the coordinate system of the VR device.

[0039] In this embodiment, the distance between the VR base station and the robotic arm is: , among which, (V x V y V z ) represents the workspace dimensions of the robotic arm, θ represents the field of view of the VR base station, and R arm The maximum turning radius of the robotic arm; and it needs to satisfy D opt ≤L max If / k is not satisfied, a larger L needs to be used. max VR base station; L max is the maximum effective positioning distance of the VR base station, k is the redundancy coverage coefficient, and the overlapping coverage coefficient of multiple base stations to ensure positioning stability (1.2-1.5).

[0040] In this embodiment, the optimal number of VR base stations is: Nopt ≥4 indicates that the robotic arm is deployed around the perimeter, ensuring no blind spots in the horizontal direction and meeting the requirement of "distributed and uniform deployment"; N opt ≥ This indicates overlapping coverage of the field of view, with 0.8 as the overlap coefficient, ensuring that the field of view of adjacent base stations overlaps by 20% to avoid blind spots; For single base station positioning accuracy, The positioning error threshold required for teaching is determined by the system's positioning accuracy requirements.

[0041] By deploying VR base stations using the optimal number and distance formula, the workspace can be covered without blind spots. Multi-view laser time difference calculation combined with redundant coverage coefficient ensures that the positioning error can accurately match the teaching requirements.

[0042] The controller, designed based on the Unity platform, communicates with the robotic arm via TCP / IP using a wired connection and with the VR locator and teaching pen via Bluetooth. The controller receives teaching trajectory pose data collected by the teaching pen and processes the data based on a preset coordinate system transformation algorithm. This maps the pen tip pose data to the robotic arm's TCP data. Specifically, by calling the coordinate transformation matrix from the VR device coordinate system 10 (obtained through hand-eye calibration) to the robotic arm base coordinate system 8, the pen tip pose data in the VR device coordinate system 10 is sequentially transformed to the robotic arm base coordinate system 8. The pen tip pose in the robotic arm base coordinate system 8 is then used as the control target for controlling the robotic arm's TCP motion. Furthermore, the controller automatically generates corresponding TCP control command arrays based on the control target dataset and publishes them to the robotic arm in a time sequence. This achieves the task of processing teaching data, publishing corresponding control targets, and reproducing the trajectory of the robotic arm based on the teaching pen pose.

[0043] The controller is designed based on the Unity platform and can be deployed on different platforms, such as PCs with Windows, Linux or Mac operating systems, and mobile devices with Android operating systems, such as mobile phones and tablets.

[0044] Taking a teaching control system designed based on the Unity platform and running on a mobile phone as an example, Figure 7 This is the initialization interface for a teaching control system designed based on the Unity platform, running as an app on a mobile device. Figure 8 For joystick control UI interface, Figure 9 For data visualization and reading interface, Figure 10 UI interface for hand-eye calibration function.

[0045] The controller and robotic arm communicate via TCP / IP Socket protocol. Unity acts as the client, connecting to the robotic arm controller's IP address and port, and wirelessly communicating with the VR positioning device and teaching pen via Bluetooth. The controller features rich functionality and a user interface; its functional framework is as follows: Figure 6 As shown, the specific functions include data and 3D model visualization, centralized management of VR device and robotic arm connection status, visualization of TCP spatial pose and teaching pen tip pose, realization of digital twins of VR device and robotic arm models, and real-time updates of the pose status of each joint of the robotic arm, each VR device, and the teaching pen; robotic arm joystick control function, which can realize six degrees of freedom control of the robotic arm TCP in the Cartesian coordinate system (X,Y,Z,RX,RY,RZ), and joint angle control of each joint of the robotic arm (joint1,joint2,joint3,joint4,joint5,joint6) can be achieved by sliding buttons; and publishing target point control of robotic arm movement, by inputting the six degrees of freedom pose of the target point through the UI interface and selecting the desired robotic arm control method, which can realize the direct publication of control target to the host computer control system to achieve robotic arm control. The automated hand-eye calibration function uses software to automate data acquisition. After executing the hand-eye calibration script, the robotic arm automatically collects 15 frames of data distributed throughout the workspace for hand-eye calibration. Each frame includes two parts of pose information: the pose data of the robotic arm TCP at a certain point in the coordinate system of the robotic arm base (X, Y, Z, RX, RY, RZ, where Cartesian coordinates represent position and axis angles represent rotation) and the pose data of the VR positioner installed at the same position on the tip of the teaching pen in the coordinate system of the VR device (X, Y, Z, PX, PY, PZ, PW). The robotic arm TCP pose data is obtained by directly parsing the data packets returned by the robotic arm. The teaching pen tip pose data is obtained by first obtaining the pose of the VR positioner installed on the VR teaching pen tip through the VR positioning system, and then obtaining the teaching pen tip pose by rigidly transforming this pose (it should be noted that the rigid transformation from the VR positioner to the tip of the teaching pen is obtained through a CAD 3D model). The collected 15 frames of data were first transformed to obtain the input data formats required by the OpenCV hand-eye calibration library: R_base2tcp, t_base2tcp, R_tracker2cam, and t_tracker2cam. R_base2tcp is a 3×3 matrix representing the rotation transformation from the robot arm base coordinate system to the robot arm TCP coordinate system, and t_base2tcp is a 1×3 matrix representing the translation transformation from the robot arm base coordinate system to the robot arm TCP coordinate system. The other data formats are processed similarly. Then, according to the eye-outside-hand calibration type, the OpenCV library functions are called to solve the equation AX=XB using the Tsai algorithm. Where (0) represents the previous frame of data, tcp represents the robot arm's tcp coordinate system, and base represents the robot arm's base coordinate system. (1) represents the transformation matrix from the robot arm base coordinate system to the center of the robot arm end tool in the previous frame of data. (2) represents the data in this frame. (3) represents the VR device coordinate system and (4) represents the VR locator coordinate system. This represents the transformation matrix from the VR locator coordinate system to the VR device coordinate system in the current frame output. Here, X is obtained by solving for X. This involves transforming the VR device coordinate system to the robotic arm base coordinate system. Teaching data processing and control command issuance: After hand-eye calibration, the pose data of the teaching pen tip in the VR device coordinate system is input. The calibration results are then used to calculate the coordinate transformation of the teaching data from the VR device coordinate system to the robotic arm base coordinate system. The specific formulas are p_base = R_cam2base × p_cam + t_cam2base, R_tcp = R_cam2base × R_cam. After the coordinate transformation, the pose of the teaching pen tip in the robotic arm base coordinate system is obtained. This data is used as the control target to be reproduced by the robotic arm TCP to issue control commands, realizing the mapping from the teaching pen's teaching data to the robotic arm TCP control target. Control commands are constructed using the pose data obtained from the coordinate transformation (X, Y, Z, RX, RY, RZ, Cartesian coordinates represent position, and axis angles represent rotation), thus realizing the issuance of robotic arm TCP control commands.

[0046] The VR-based robotic arm teaching system provided in this embodiment is as follows: Figure 11 As shown, its working steps are as follows: S1: Install VR Positioner: Install a VR positioner at the end joint of the robotic arm using a 3D-printed firmware made of polymer material, ensuring that its sensor faces upward to reduce sensor obstruction and ensure accurate positioning. This VR positioner acts like a calibration target during the hand-eye calibration stage and provides pose feedback during teaching control to improve the control accuracy of the robotic arm.

[0047] In S1, a VR positioner is installed on the end joint of the robotic arm to locate the position of the end effector. The accurate TCP tip position needs to be calculated by rigid transformation from coordinate system A to coordinate system B. The rigid transformation from coordinate system A to coordinate system B is obtained by translation and rotation transformation calculated based on the CAD model.

[0048] S2: Deploy the VR positioning system: Build a positioning system based on VR devices, install and debug multiple VR base stations 7 to ensure coverage of the workspace; the teaching pen is connected to the controller via Bluetooth, starts the automatic calibration program, and enables the VR locator to automatically update its pose and complete the device status calibration based on the laser signal.

[0049] In S2, the specific steps include: Place 2-4 VR base stations based on the robotic arm's workspace, the teaching area required for the robotic arm's operation, and potential occlusion of the VR locator. The VR base stations should be deployed as evenly as possible around the workspace. The VR positioning system is compatible with multiple VR base station devices for simultaneous positioning; the more VR base station devices used, the higher the positioning accuracy. The VR base stations have a built-in automatic relative position calibration program. By repeatedly moving the VR locator within the positioning space, the relative position of the VR base stations can be calibrated, and the visualization model in Unity can be updated.

[0050] S3: Establish coordinate systems and acquire data: Design the controller and establish multiple tool coordinate systems (such as the VR positioner coordinate system, the robotic arm base coordinate system, etc.); acquire the pose data of the VR positioner in different coordinate systems through the VR positioning system, and calculate the rigid transformation relationship between the coordinate systems by combining the data of the robotic arm and the CAD model.

[0051] In S3, a VR locator is installed on the tip of the teaching pen to locate the pose of each point during the teaching process. The accurate pose of the teaching pen tip needs to be calculated by rigid transformation from the coordinate system tracker2 of the VR locator installed on the teaching pen tip to the coordinate system pen of the teaching pen tip. The rigid transformation from the tracker2 coordinate system to the pen coordinate system can be obtained by calculating translation and rotation transformation based on the CAD model. The pose of the VR locator in the VR device system coordinate system camera is read by the VR locator, and the pose of the pen tip in the VR device coordinate system is calculated by rigid transformation as the initial teaching data Pcam.

[0052] It should be noted that, as Figure 4 As shown, the teaching pen contains two coordinate systems: one is the VR locator coordinate system located at the pen tip, and the other is the pen tip coordinate system located at the pen tail. Here, the VR locator pose is captured using a VR positioning system. The captured pose is the VR locator's pose in the VR device coordinate system (camera). Hand-eye calibration requires inputting pen tip pose data, but the teaching pen can only measure VR locator data. Therefore, a rigid transformation needs to be added, i.e., a rigid transformation from the VR device locator at the pen tip to the pen tip coordinate system at the pen tail. The pen tip pose in the VR device coordinate system is used as the initial teaching data Pcam.

[0053] S4: Perform hand-eye calibration: Press the "hand-eye calibration" button on the teaching pen, and the controller will automatically run the calibration program, collect 15 frames of hand-eye calibration data in spatial position, and calculate the coordinate transformation between the VR device coordinate system and the robotic arm base coordinate system.

[0054] S4 specifically includes the following steps: First, press the "Hand-Eye Calibration" button on the teaching pen to run the automatic hand-eye calibration program designed by the controller. Collect 15 frames of hand-eye calibration input data. Each frame contains two inputs: the pose of the robotic arm TCP in the robotic arm base coordinate system (data format: X, Y, Z, RX, RY, RZ, using Cartesian coordinates to represent position and axis angles to represent rotation, with units of millimeters and radians respectively) and the pose of the VR positioner installed at the end joint of the robotic arm in the VR device camera coordinate system (data format: X, Y, Z, PX, PY, PZ, PW, using Cartesian coordinates to represent position and quaternions to represent rotation). The data is automatically collected through the UI interface buttons to ensure that the TCP pose and the teaching pen tip pose of each frame come from the same timestamp.

[0055] Next, the collected data is stored in a pre-defined raw data array for hand-eye calibration. An automated script automatically performs pre-checks on the raw data, verifying that there are at least four data sets. It also automatically performs data format conversion, unit conversion, quaternion normalization, and generates a homogeneous transformation matrix. Finally, it automatically performs coordinate transformations and inversions to obtain R_base2tcp, t_base2tcp, R_tracker2cam, and t_tracker2cam. R_base2tcp represents the rotation transformation from the robot's base coordinate system (base) to the TCP coordinate system (tcp); t_base2tcp represents the translation transformation from the robot's base coordinate system (base) to the TCP coordinate system (tcp); R_tracker2cam represents the rotation transformation from the VR locator coordinate system (Tracker) to the VR device coordinate system (camera); and t_tracker2cam represents the translation transformation from the VR locator coordinate system (Tracker) to the VR device coordinate system (camera).

[0056] Finally, the following equation is solved by calling the OpenCV hand-to-eye calibration library function according to the Hand-to-Eye calibration type: R_base2tcp×R_cam2tcp=R_cam2base×R_tracker2cam; R_base2tcp×t_cam2tcp+t_base2tcp=R_cam2base×t_tracker2cam+t_cam2base; The Tsai two-step solution method first solves the rotation part and then the translation part, finally obtaining the transformation matrix T_cam2base from the VR device coordinate system camera to the robotic arm base coordinate system base.

[0057] S5: Draw teaching trajectory: After completing hand-eye calibration, press the "Collect Data" button on the teaching pen to draw the teaching trajectory in the workspace and record the pose data of the teaching pen at each discrete point.

[0058] S5 specifically includes the following steps: The teaching trajectory data is collected using a teaching pen. The specific operation is as follows: Press and hold the "Start Collection" button on the teaching pen and keep the button pressed. At the same time, hold the teaching pen and draw the teaching trajectory along the spatial path of the desired TCP tip movement. The time from pressing the "Start Collection" button to releasing it is recorded as one teaching trajectory data. Each segment of teaching trajectory data includes the pose pcam of the teaching pen tip at each discrete point. Based on the calculated pbase, the control target for controlling the movement of the robotic arm TCP is obtained (data format is X, Y, Z, RX, RY, RZ, using Cartesian coordinates to represent position and axis angles to represent rotation, with units of millimeters and radians respectively). Thus, the pen tip pose in the VR device's camera coordinate system is used as the teaching data input to realize the reproduction of the teaching pen tip pose of the robotic arm TCP in the base coordinate system base.

[0059] S6: Execute trajectory planning: After data acquisition is completed, press the "Plan and Execute" button. The controller processes the teaching trajectory data and generates control commands. The robotic arm reproduces the movement of the teaching pen according to the acquired trajectory, realizing precise control of the robotic arm's TCP end effector.

[0060] S6 specifically includes the following steps: After data acquisition is complete, press the "Plan and Execute" button on the teaching pen to input the collected teaching trajectory data into the predefined sequence space of the software. The translation transformation t_cam2base and rotation transformation R_cam2base are extracted from the transformation matrix T_cam2base from the VR device coordinate system camera obtained by hand-eye calibration to the robot arm base coordinate system base. The pose pbase of the teaching pen after coordinate transformation in the robot arm base coordinate system is calculated by the mathematical formulas: p_base=R_cam2base×p_cam+t_cam2base, R_tcp=R_cam2base×R_tracker×R_tracker2tcp. Based on the calculated pbase, it serves as the control target for controlling the movement of the robotic arm TCP (data format: X, Y, Z, RX, RY, RZ, using Cartesian coordinates to represent position and axis angles to represent rotation, with units of millimeters and radians respectively). Thus, the pen tip pose in the VR device's camera coordinate system is used as the teaching data input to control the robotic arm TCP in the base coordinate system base to reproduce the teaching pen tip pose.

[0061] According to the above process, the poses of all data points are sequentially transformed to generate the desired TCP pose data under the robot arm base coordinate system. The generated robot arm TCP control commands are then stored in a predefined sequence space in the software. The packaged control command set is sent to the robot arm's lower-level computer at once to control the robot arm's end effector to reproduce the trajectory of the teaching pen tip. By continuously repeating the processes described in S5 and S6, the desired trajectory can be drawn using the teaching pen, and then commands can be sent to control the robot arm's TCP to reproduce the desired trajectory, thus completing the robot arm teaching task.

[0062] The VR-based robotic arm teaching system provided in this embodiment uses a teaching pen and a VR positioner at the end effector (TCP) of the robotic arm to reverse-position itself and input the data into a controller. The controller processes the raw pose data sent by the VR positioner, performing coordinate transformations and other data processing to obtain the robotic arm TCP control target in the robotic arm base coordinate system. The controller then sends control commands to quickly control the robotic arm TCP to reproduce the spatial trajectory drawn by the teaching pen. The control platform is developed based on Unity and can realize functions such as data acquisition, data processing, hand-eye calibration, visualization, and sending control targets. The controlled robotic arm is a six-axis multi-degree-of-freedom robotic arm, and its end effector (TCP) is equipped with end effectors such as a welding torch for use in welding, gluing, and other work areas.

[0063] The VR-based robotic arm teaching system provided in this embodiment enables rapid and efficient control of the robotic arm's TCP to reproduce the spatial motion trajectory drawn by the teaching pen. It achieves high-precision and smooth trajectory teaching data transmission and analysis, improving the efficiency and ease of operation of TCP end-position pose teaching.

[0064] The VR-based robotic arm teaching system provided in this embodiment transforms complex robotic arm trajectory programming into an intuitive physical action of "drawing in the air with a handheld teaching pen," abandoning the traditional methods of teaching pendant inching or code writing. This novel teaching device, the teaching pen, makes the worker's teaching process similar to drawing with a pen in daily life, allowing workers to easily learn and operate the system without complex training. This handheld teaching device enhances the freedom of human-computer interaction and collaboration. By replacing traditional programmatic teaching with handheld device teaching, it effectively improves the efficiency of robotic arm teaching in production and daily life, lowers the learning threshold for robotic arm teaching, increases the flexibility of robotic arm teaching, and makes the teaching process more intuitive.

[0065] The VR-based robotic arm teaching system provided in this embodiment utilizes a VR positioning system and a supporting Unity-based control system, which offer greater flexibility and adaptability. Firstly, the VR positioning system can be flexibly deployed according to the workspace. By flexibly deploying the VR positioning base station locations and adjusting the number of VR positioning base stations, it can effectively handle workspaces of different sizes and shapes. Furthermore, the VR positioning base stations can automatically calibrate their relative spatial poses at regular intervals, enabling accurate repositioning of each base station after movement. Secondly, the supporting Unity-based control system has excellent cross-platform compatibility and can be easily deployed on various hardware devices. The teaching pen button and the control system UI allow for quick triggering of different functions, adapting to diverse teaching scenario requirements.

[0066] The VR-based robotic arm teaching system provided in this embodiment effectively improves teaching accuracy and adaptability to teaching scenarios by leveraging a VR device positioning system. On one hand, this invention employs multiple VR positioning base stations to locate the VR locator from different positions. Compared to a single-camera positioning system, this effectively reduces positioning failures caused by material obstructions from the robotic arm and environment, improving the stability and adaptability of the positioning system. This allows the robotic arm to maintain accurate positioning even in complex working environments, making it suitable for teaching scenarios with potential obstructions, such as welding and gluing. On the other hand, by introducing a high-precision VR laser positioning system, this invention can capture the six-degree-of-freedom spatial pose of the teaching pen tip with sub-millimeter precision. Combined with hand-eye calibration and coordinate system transformation, the trajectory data is accurately mapped to the robotic arm base coordinate system. This effectively avoids problems such as uneven trajectory and large cumulative accuracy errors caused by manual approximation in traditional teaching, ensuring the robotic arm end effector faithfully and smoothly reproduces the teaching trajectory.

[0067] Example 2 This embodiment provides a control method for a VR-based robotic arm teaching system as described in Embodiment 1.

[0068] The control method for the VR-based robotic arm teaching system provided in this embodiment, as described in Embodiment 1, includes the following steps: Step 1: Install a VR locator on the end joint of the robotic arm and fix the VR locator to the end joint using a 3D-printed firmware.

[0069] In step 1, the VR locator is connected to the end joint of the robotic arm via a 3D-printed firmware. The sensor protrusion of the VR locator faces away from the TCP end of the robotic arm, so that the sensor of the VR locator faces upwards during most operations. This greatly reduces the blind spots caused by the robotic arm joint obstructing the sensor, which would prevent accurate positioning of the VR locator's pose. This can enhance the reliability of positioning information acquisition during the operation of the robotic arm and lay a reliable foundation for subsequent accurate positioning of the robotic arm TCP in the workspace.

[0070] In step 1, the VR positioner installed on the end effector joint of the robotic arm mainly serves two purposes: Firstly, during the hand-eye calibration stage, the VR positioner acts as a calibration target in the traditional hand-eye calibration principle. During calibration, the VR positioner moves with the robotic arm TCP in the workspace, and the VR positioning system determines the VR positioner's pose in the VR device coordinate system. Then, through a rigid transformation from the VR positioner to the robotic arm TCP, the pose of the robotic arm TCP in the VR device coordinate system is obtained. Finally, data from the lower-level robotic arm machine is used to obtain the pose of the robotic arm TCP in the base coordinate system. This yields the pose of the robotic arm in both the VR device coordinate system and the robotic arm base coordinate system at the same absolute position, which is used to establish the subsequent hand-eye calibration equations. Secondly, during subsequent teaching and control, the pose output by the VR positioner can serve as feedback information to correct the robotic arm's control target, further improving the robotic arm's control accuracy.

[0071] Step 2: Establish a VR-based positioning system. Distribute VR positioning base stations around the robotic arm's workspace to ensure that all working areas are covered by the VR positioning base station's illumination range. After deploying the VR positioning base stations, power on the teaching pendant equipped with the VR locator and connect it to the computer via Bluetooth. Then, hold the teaching pendant with the VR locator and make simple movements in the workspace to trigger the VR positioning base station's automatic calibration program, automatically completing the relative pose calibration of the VR device.

[0072] In step 2, the number and installation location of VR positioning base stations can be flexibly adjusted according to the size and shape of the workspace, ensuring that all areas in the workspace are directly illuminated by laser signals emitted from at least two base stations simultaneously. A minimum of two VR positioning base stations are used in the VR positioning system; the more VR positioning base stations used, the higher the positioning stability of the VR positioning device. VR positioning base stations are generally mounted on tripods, and the installation height can be flexibly adjusted according to the workspace. In this embodiment, the VR positioning base stations are installed approximately 0.5 meters above the workspace, with the base stations tilted downwards at a 30-degree angle from the vertical plane.

[0073] In step 2, after deploying the VR positioning base stations and powering on all base station devices, the VR locator is moved at a constant speed or waved back and forth along a trajectory within the workspace. The VR locator will use the laser angle received by the sensor to determine its own position. The base station devices will automatically update the relative pose information of the base station devices in space according to the mounted automatic calibration script. The relative pose of each VR positioning base station in the VR device coordinate system is calibrated through spatial geometric algorithms, and the device status information is sent to the background control system via wireless Bluetooth to realize automated management of the device status of the VR positioning system and achieve precise positioning of the VR locator.

[0074] Step 3: The controller will establish multiple tool coordinate systems, namely the VR device coordinate system camera, the robot arm base coordinate system base, the teaching pen tip coordinate system pen, the robot arm TCP coordinate system tcp, the VR locator coordinate system tracker1 installed on the end joint of the robot arm, and the VR locator tracker2 installed on the tip of the teaching pen; the pose information of each coordinate system can be directly read by the robot arm or VR positioning system, or indirectly calculated through rigid transformation.

[0075] In step 3, the pose data of VR locator coordinate systems tracker1 and tracker2 in the VR device coordinate system camera can be obtained through wireless Bluetooth data from the VR positioning system. The pose data of the robotic arm TCP coordinate system tcp in the robotic arm base coordinate system can be obtained through data transmitted back via the robotic arm's TCP / IP protocol. Using the CAD 3D model of the robotic arm with the VR locator installed, the rigid transformation from the VR locator coordinate system tracker1 to the robotic arm TCP coordinate system tcp can be calculated. Using the CAD 3D model of the teaching pen, the rigid transformation from the VR locator coordinate system tracker2 to the teaching pen tip coordinate system pen can be calculated. By performing a rigid transformation on the poses of the VR locator coordinate systems tracker1 and tracker2 in the VR device coordinate system camera, the poses of the robotic arm TCP coordinate system tcp and the teaching pen tip coordinate system pen in the VR device coordinate system can be obtained.

[0076] Step 4: Press the "Hand-Eye Calibration" button on the teaching pen, and the controller will automatically run the hand-eye calibration script. The hand-eye calibration program calculates the coordinate transformation from the VR device coordinate system (camera) to the robotic arm base coordinate system (base), thus linking the VR device coordinate system and the robotic arm TCP coordinate system.

[0077] In step 4, the automated hand-eye calibration process involves the robotic arm TCP automatically moving to 15 evenly distributed spatial positions in the workspace. At each position, one frame of hand-eye calibration data is collected, for a total of 15 frames. Each frame includes two parts: the pose of the robotic arm TCP in the robotic arm base coordinate system (base) and the pose of the VR positioner mounted on the robotic arm in the VR device coordinate system (camera). The pose data format of the robotic arm TCP in the robotic arm base coordinate system (base) is (X, Y, Z, RX, RY, RZ), with the first three digits representing position in Cartesian coordinates and the last three digits representing rotation in axis-angle coordinates. The pose data format of the VR positioner mounted on the robotic arm in the VR device coordinate system (camera) is (X, Y, Z, PX, PY, PZ, PW), with the first three digits representing position in Cartesian coordinates and the last four digits representing rotation in quaternions. The input data is format-transformed to obtain R_base2tcp, t_base2tcp, R_tracker2cam, and t_tracker2cam. The OpenCV library function is then called to perform hand-eye calibration. The coordinate transformation from the VR device coordinate system (camera) to the robotic arm base coordinate system (base) is calculated and saved to a predefined storage array for subsequent coordinate transformation calls.

[0078] Step 5: Using a teaching pen equipped with a VR locator, draw the teaching trajectory in the robotic arm's workspace as needed, specifically: After ensuring the completion of step 4 (hand-eye calibration), hold the teaching pen and press the "Acquire Data" button to begin capturing the teaching trajectory. Simultaneously, use the teaching pen to draw the teaching trajectory in the workspace as needed. Release the "Acquire Data" button to end the drawing process. The controller will save the teaching trajectory data during the button press process to a predefined array. The teaching trajectory data records the pose data of the teaching pen at various discrete points during the drawing process.

[0079] Step 6: After completing the data acquisition step in Step 5, press the "Plan and Execute" button on the teaching pen to process the captured teaching trajectory and issue control commands. After receiving the control command data packet, the robotic arm will reproduce the pose of each discrete point on the trajectory drawn by the teaching pen during the data acquisition stage. This completes the acquisition of the teaching trajectory by the teaching pen for the implementation of TCP end control of the robotic arm.

[0080] In step 6, the captured teaching trajectory is processed and control commands are issued as follows: The controller inputs the predefined input data array from step 5, reads the pose information of the VR locator installed on the teaching pen tip in the VR device coordinate system camera, obtains the pose information of the teaching pen tip in the VR device coordinate system camera through rigid transformation, calls the hand-eye calibration results from step 4, performs coordinate transformation on the pose information in the VR device coordinate system camera to obtain the pen tip pose data in the robot arm base coordinate system base, uses the pose information of the teaching pen tip in the robot arm base coordinate system base as the control target of the mapped teaching trajectory, and automatically edits and issues control commands to the robot arm communication protocol to realize the robot arm TCP teaching control.

[0081] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A VR-based teaching system for a robot arm, characterized in that: The teaching pen, the mechanical arm, the controller and a plurality of VR base stations are included, and the VR locator is arranged on the teaching pen. The VR base stations are uniformly distributed around the working area of the mechanical arm, and the VR locator is irradiated by laser beams of different angles. The VR locator is used for receiving laser beams of different angles and inversely positioning the pose of the teaching pen in the VR device coordinate system. The controller is used for reading the pose of the teaching pen in the VR device coordinate system, calling the hand-eye calibration result, performing coordinate transformation on the pose in the VR device coordinate system, obtaining the pose of the teaching pen in the base coordinate system of the mechanical arm, taking the pose of the teaching pen in the base coordinate system of the mechanical arm as a mapping result to obtain a teaching trajectory control target, editing and publishing a control instruction into a communication protocol of the mechanical arm, and realizing teaching control of the mechanical arm.

2. The VR-based robotic arm teaching system of claim 1, wherein: The logic output circuit of the teaching pen is connected with a plurality of metal spring needle pins, and different event signals are sent to the controller when the pins of different groups of metal spring needles are short-circuited.

3. The VR-based robotic arm teaching system of claim 1, wherein: The distance between the VR base station and the mechanical arm is: where (V x , V y , V z ) is the mechanical arm workspace size, θ is the VR base station field of view angle, R arm is the maximum turning radius of the mechanical arm.

4. The VR-based robotic arm teaching system of claim 1, wherein: The distance between the VR base station and the mechanical arm satisfies D opt ≤L max / k, if not, replace a larger L max VR base station, wherein L max is the maximum effective positioning distance of the VR base station, and k is a redundancy coverage coefficient.

5. The VR-based robotic arm teaching system of claim 1, wherein: The optimal number of VR base stations is ; wherein θ is the field of view angle of the VR base station, is the positioning accuracy of a single base station, is the positioning error threshold.

6. The VR-based robotic arm teaching system of claim 1, wherein: The VR locator is connected with the joint at the end of the mechanical arm through 3D printing, and the sensor protrusion of the VR locator is opposite to the TCP end of the mechanical arm.

7. A control method of a VR-based robot teaching system, characterized by: The VR-based teaching system of the mechanical arm is suitable for the VR-based teaching system of the mechanical arm according to any one of claims 1-6, and includes: reading the pose of the teaching pen in the VR device coordinate system, calling the hand-eye calibration result, performing coordinate transformation on the pose in the VR device coordinate system, and obtaining the pose of the teaching pen in the base coordinate system of the mechanical arm; taking the pose of the teaching pen in the base coordinate system of the mechanical arm as a mapping result to obtain a teaching trajectory control target, editing and publishing a control instruction into a communication protocol of the mechanical arm, and realizing teaching control of the mechanical arm.

8. The control method according to claim 7, characterized by: In the hand-eye calibration process, the VR locator mounted on the mechanical arm moves with the TCP of the mechanical arm in the working area, the pose of the VR locator mounted on the mechanical arm in the VR device coordinate system is positioned, then the pose of the TCP of the mechanical arm in the VR device coordinate system is obtained through rigid transformation from the VR locator mounted on the mechanical arm to the TCP of the mechanical arm, the pose of the TCP of the mechanical arm in the base coordinate system is obtained through the lower computer of the mechanical arm, and the poses of the mechanical arm in the VR device coordinate system and the base coordinate system of the mechanical arm at the same absolute position are obtained.

9. The control method according to claim 8, characterized by: The rigid transformation is calculated through the CAD three-dimensional model of the VR locator mounted on the mechanical arm.

10. The control method of claim 7, wherein: The Tsai algorithm is used to realize the hand-eye calibration according to the calibration type of the eye outside the hand.

Images