A mixed reality-based robot teaching method

Abstract

A mixed reality-based robotic arm teaching method, belonging to the field of robot teaching technology, includes the following steps: interacting with a mixed reality virtual robotic arm in mixed reality glasses: rotating the virtual robotic arm joints and dragging the virtual robotic arm end effector, and selecting a robotic arm teaching mode; in the mixed reality glasses, according to different teaching modes, the glasses acquire different data and output them to the real robotic arm; after receiving the joint data, the real robotic arm controller drives the real robotic arm to complete the same movement; the real robotic arm controller smooths the pose data of the virtual robotic arm end effector relative to the base and inputs it into an inverse kinematics algorithm to solve for the joint angle parameters of the real robotic arm; during Cartesian space teaching, the virtual robotic arm produces the same movements as the real robotic arm, thereby completing the teaching. This application, through mixed reality and simulation, greatly reduces safety issues during teaching while improving teaching efficiency.

Description

Technical Field

[0001] This invention relates to teaching methods, and more particularly to a robotic arm teaching method based on mixed reality, belonging to the field of robot teaching technology. Background Technology

[0002] Robotic arms can replace humans in performing dangerous and complex operations in harsh environments, and are more adaptable and efficient at repetitive, single tasks than human workers. However, the level of intelligence in current multi-joint robotic arms is not high; completing a specified motion process usually requires human instruction, and the efficiency of the motion needs to be improved.

[0003] Currently, there are two main teaching methods for multi-joint serial robotic arms (hereinafter referred to as robotic arms): one is online teaching, where the operator controls the robotic arm to move in joint space and Cartesian space through a wired teach pendant, and to move to a designated point with a specified trajectory and posture to complete the task. This teaching method allows the operator to observe the movement of the robotic arm in real time and adjust its trajectory accordingly. However, the operator cannot directly observe the target position at the end of the robotic arm, and there is close interaction between the operator and the robotic arm itself. In case of danger, the operator's personal safety will be seriously threatened. Moreover, the entire teaching process is cumbersome and can easily lead to rigid movements, greatly reducing overall efficiency. The other method is offline teaching, which uses offline programming software based on a simulator to teach the robotic arm. Although this teaching method avoids the safety hazards caused by close interaction between the operator and the robotic arm, it is difficult to use due to problems such as inaccurate environmental and object models, complex working scenarios, and many human interaction links.

[0004] In actual production work, frontline employees, due to their long-term work experience, have accumulated rich operational expertise and can summarize the technical essentials and operational taboos for efficiently completing a certain task. Therefore, how to enable operators to teach robotic arms step by step based on their own operational characteristics, and achieve the efficient transfer of skills and experience from humans to robotic arms, is one of the main research directions at present. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes a robotic arm teaching method based on mixed reality. This method combines mixed reality technology with simulation, which greatly reduces safety issues during teaching and improves teaching efficiency. By combining mixed reality with robotic arm teaching algorithms, the system has a better immersive environment.

[0006] A mixed reality-based robotic arm teaching method includes the following steps:

[0007] Step S1: In the mixed reality glasses, interact with the mixed reality virtual robotic arm: rotate the joints of the virtual robotic arm and drag the end effector of the virtual robotic arm, and select the robotic arm teaching mode: joint space teaching and Cartesian space teaching;

[0008] Step S2: In the mixed reality glasses, according to different teaching modes in step S1, the glasses acquire different data and output them to the real robotic arm: if step S1 is joint space teaching, the glasses acquire the angle data of each joint of the virtual robotic arm and output the data; if step S1 is Cartesian space teaching, the glasses continuously acquire and output the pose data of the virtual robotic arm end tool relative to the base, and wait for the real robotic arm controller to input data.

[0009] Step S3: After receiving the joint data from step S2, the real robotic arm controller drives the real robotic arm to complete the same movement. The real robotic arm controller smooths the pose data of the virtual robotic arm end tool relative to the base from step S2, and then inputs it into the inverse kinematics algorithm to solve the joint angle parameters of the real robotic arm. In this way, the real robotic arm is driven to complete the corresponding movement while the joint angle is output to the glasses.

[0010] Step S4: During Cartesian space teaching, the glasses use the joint angle data output in step S3 to synchronously update and draw the virtual robotic arm, so that the virtual robotic arm produces the same movements as the real robotic arm, thereby completing the teaching.

[0011] The advantages of this invention compared to the prior art are:

[0012] This invention combines mixed reality technology with a more intuitive and natural teaching method, taking into account the characteristics of the operator's work. It allows for hands-on teaching of the robotic arm while ensuring the operator's safety, achieving efficient skill transfer from human to robotic arm. Compared to traditional robotic arm teaching, the use of the currently popular mixed reality technology significantly reduces safety issues during teaching in a structured work environment, while simultaneously improving teaching efficiency, providing a novel solution for robotic arm task teaching deployment. Furthermore, combining mixed reality with robotic arm teaching algorithms creates a more immersive environment, and drag-and-drop teaching allows for rapid movement of the robotic arm to the teaching point, simplifying the operation process and improving the ease of human-computer interaction.

[0013] The technical solution of this application will be further described below with reference to the accompanying drawings and embodiments: Attached Figure Description

[0014] Figure 1 This invention application includes a flowchart of a robotic arm teaching method based on mixed reality.

[0015] Figure 2 A schematic diagram of the coordinate system of the virtual robotic arm base and the coordinate system of the real robotic arm base;

[0016] Figure 3 A teaching system diagram used in a specific embodiment of the teaching method;

[0017] Figure 4 A curve showing the joint position of the virtual robotic arm when the stiffness coefficient of the spring model is set to a fixed value;

[0018] Figure 5 The velocity curve of the virtual robotic arm joint when the stiffness coefficient of the spring model is set to a fixed value;

[0019] Figure 6 The curve of the joint position of the virtual robotic arm when the stiffness coefficient of the spring model is taken as the function described in equation (3);

[0020] Figure 7 The virtual robotic arm joint velocity curve when the spring model stiffness coefficient is taken as the function described in equation (3);

[0021] Figure 8 A graph showing the joint positions of the robotic arm during mixed reality simulation teaching;

[0022] Figure 9 This is a graph showing the joint velocity curves of a robotic arm during mixed reality simulation teaching. Detailed Implementation

[0023] The technical solution of this application will now be described in detail with reference to the accompanying drawings and embodiments. Unless otherwise stated, the technical or scientific terms used in this application have the ordinary meaning as understood by those skilled in the art.

[0024] Reference Figures 1-3 This embodiment provides a robotic arm teaching method based on mixed reality. The teaching system combined with this embodiment is as follows: Figure 3 The image includes: 1. a real robotic arm, 2. a robotic arm controller, 3. a virtual robotic arm, 4. mixed reality glasses, and 5. an operator.

[0025] The method includes:

[0026] Step S1: In the mixed reality glasses 4, interact with the mixed reality virtual robotic arm in different ways: rotate the joints of the virtual robotic arm and drag the end effector of the virtual robotic arm, and select the robotic arm teaching mode: joint space teaching and Cartesian space teaching;

[0027] Step S2: In the mixed reality glasses 4, according to different teaching modes in step S1, the glasses acquire different data and output them to the real robotic arm 1: If step S1 is joint space teaching, the glasses acquire the angle data of each joint of the virtual robotic arm 3 and click the mixed reality button to output the data; if step S1 is Cartesian space teaching, the glasses continuously acquire and output the pose data of the virtual robotic arm end tool relative to the base, and wait for the real robotic arm controller 2 to input data;

[0028] Step S3: After receiving the joint data from step S2, the real robotic arm controller 2 drives the real robotic arm 1 to complete the same movement. The real robotic arm controller 2 smooths the pose data of the virtual robotic arm end tool relative to the base from step S2, and then inputs it into the inverse kinematics algorithm to solve the joint angle parameters of the real robotic arm. In this way, the real robotic arm 1 is driven to complete the corresponding movement while the joint angle is output to the glasses.

[0029] Step S4: During Cartesian space teaching, the glasses use the joint angle data output in step S3 to synchronously update and draw the virtual robotic arm 3, so that the virtual robotic arm 3 produces the same movements as the real robotic arm 1, thereby completing the teaching.

[0030] In step S1, after the operator 5 puts on the HoloLens2 mixed reality glasses, the glasses perceive the surrounding environment and determine their own position through their built-in camera and sensors. Then, the virtual robotic arm and part of the virtual teaching scene are integrated into the mixed reality environment and presented to the user. At the same time, the user's gestures, posture, voice and other information are recognized to realize the interaction between the user and the virtual robotic arm, including adjusting the position of the virtual robotic arm to overlap with the real robotic arm, rotating the joints of the virtual robotic arm, dragging the end of the virtual robotic arm, selecting the robotic arm teaching mode, etc.

[0031] Step S2 includes the following steps:

[0032] (1) Data composition

[0033] The virtual robotic arm joint angle data includes the angle information of the six rotational joints and the opening and closing state information of the gripper. The pose data of the robotic arm end tool relative to the base includes its three-dimensional position information and three-dimensional posture information. In addition, the data also includes a flag to distinguish between the two types of data so that the controller in step S3 can execute different programs. At the same time, the flag can also be used to realize more functions without the need for additional control signals, such as ending teaching, closing communication, etc.

[0034] (2) Data conversion

[0035] Because the coordinate system o-xyz of the mixed reality environment in the glasses uses a left-handed coordinate system, while the coordinate system O-XYZ of the real environment in which the robotic arm is located uses a right-handed coordinate system, such as Figure 2 As shown, the end-effector pose data of a robotic arm will differ in different coordinate systems. If it is not converted, it will lead to incorrect robotic arm movements and pose a serious safety hazard. The communication method used also has requirements on the data type, such as floating-point numbers and bytes. The data must be converted to the appropriate type; otherwise, the robotic arm and glasses will not be able to recognize the received data, which will lead to danger.

[0036] (3) Joint angle initialization

[0037] If the Cartesian space teaching is performed in step S1, the real robotic arm initializes each joint and outputs the joint initialization angle to the mixed reality glasses. After receiving the data, the mixed reality glasses 4 updates and draws the virtual robotic arm 3, which facilitates the overlap of the virtual and real robotic arms and enhances the sense of presence. Subsequently, the mixed reality glasses 4 continuously acquires and outputs the pose data of the virtual robotic arm end tool relative to the base, and waits for the real robotic arm controller 2 to input data.

[0038] Furthermore, in step S3: the virtual robotic arm end-effector pose data is smoothed to minimize data fluctuations caused by gesture teaching, thereby minimizing fluctuations in the joint angular velocity and end-effector vibration of the real robotic arm.

[0039] Furthermore, the pose data of the virtual robotic arm's end effector is smoothed, and the smoothing algorithm is as follows:

[0040] The spring model is used as a filtering algorithm to process the position data of the virtual robotic arm's end effector. The mathematical expression of the spring model is:

[0041] T pos '=T pos +k p ·(H pos -T pos (1)

[0042] Δ=H pos -T pos (2)

[0043] Among them, T pos 'T' represents the target position of the robotic arm at that moment. pos H represents the target position of the robotic arm at the previous moment. pos The desired position of the robotic arm is represented by Δ, which is the position output by the glasses. Δ represents the tracking error, and the parameter k is... p This is the stiffness coefficient of the spring model, i.e., the proportional coefficient for tracking error compensation.

[0044] If parameter k pAs a constant, when the tracking error is large, the actual robotic arm approaches the desired position faster, and the speed of each joint is also faster. Furthermore, the larger the tracking error, the faster the actual robotic arm moves, posing a higher safety hazard. Taking a six-joint robotic arm as an example: Figure 4 and Figure 5 As shown. To solve the above problem, k is designed. p The error varies with the tracking error of the robotic arm. To address the above issues and ensure smooth joint position changes and minimal angular velocity fluctuations during fixed-position robotic arm operation, extensive simulation experiments were conducted, and a final design for k was developed. p It varies with the actual tracking error of the robotic arm:

[0045]

[0046] Wherein, parameter K is the amplitude of the sine function, and parameter a is the value of k. p The boundary value between the two trends changing with the tracking error is given by parameter b, which is the sinusoidal function offset. This ensures that when the tracking error is greater than or equal to a, the robotic arm moves towards the desired position at a safe speed of (b+K)·a, preventing the robotic arm from moving too fast and causing danger. As the tracking error decreases to a, the tracking speed of the robotic arm smoothly decreases, with the speed decreasing as the tracking error decreases. This is illustrated using a six-joint robotic arm as an example. Figure 6 and Figure 7 As shown, this avoids excessive acceleration caused by the robotic arm stopping when it reaches the desired position, greatly ensuring the safety of both equipment and personnel.

[0047] The virtual robotic arm's end-effector posture is represented using quaternions, and the posture data is processed using a spherical interpolation algorithm. The mathematical expression of spherical interpolation is:

[0048]

[0049] Wherein, parameter p represents the initial posture unit quaternion, parameter q represents the final posture unit quaternion, parameter θ represents the radian angle between the two quaternions p and q, and parameter t represents the spherical interpolation scaling factor, thus ensuring smooth changes in the virtual end-effector posture data received by the real robotic arm. Parameter t was determined through numerous simulation teaching experiments to achieve smooth changes in the joint positions of the real robotic arm and minimal fluctuations in angular velocity during teaching, using a six-joint robotic arm as an example; Figure 8 and Figure 9 As shown, this also makes the changes in the position and speed of the robotic arm's end effector smoother, improving teaching safety.

[0050] In step S4, during Cartesian space teaching, the mixed reality glasses synchronously update and draw the virtual robotic arm using the joint angle data output by the real robotic arm controller in step S3, so that the virtual robotic arm's movements are the same as the real robotic arm, making the robotic arm movements taught by mixed reality closer to reality. At the same time, mixed reality can be used to know in advance the expected position of the robotic arm's end in the real environment, further avoiding collisions between the real robotic arm and the work scene, thereby completing the teaching.

[0051] The present invention has been disclosed above with reference to preferred embodiments, but it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed structure and technical content to create equivalent embodiments without departing from the scope of the present invention, and all such modifications or alterations shall still fall within the scope of the present invention.

Claims

1. A mixed reality-based teaching method for a robot arm, characterized by, It includes the following steps: Step S1: In the mixed reality glasses, interact with the mixed reality virtual robotic arm: rotate the joints of the virtual robotic arm and drag the end effector of the virtual robotic arm, and select the robotic arm teaching mode: joint space teaching and Cartesian space teaching; Step S2: In the mixed reality glasses, according to different teaching modes in step S1, the glasses acquire different data and output them to the real robotic arm: if step S1 is joint space teaching, the glasses acquire the angle data of each joint of the virtual robotic arm and output the data; if step S1 is Cartesian space teaching, the glasses continuously acquire and output the pose data of the virtual robotic arm end tool relative to the base, and wait for the real robotic arm controller to input data. Step S3: After receiving the joint data from step S2, the real robotic arm controller drives the real robotic arm to complete the same movement. The real robotic arm controller smooths the pose data of the virtual robotic arm end tool relative to the base from step S2, and then inputs it into the inverse kinematics algorithm to solve the joint angle parameters of the real robotic arm. In this way, the real robotic arm is driven to complete the corresponding movement while the joint angle is output to the glasses. In step S3, the pose data of the virtual robotic arm end tool relative to the base is smoothed to reduce data fluctuations caused by gesture teaching, thereby reducing fluctuations in the joint angular velocity and end-effector vibration of the real robotic arm. The pose data of the virtual robotic arm's end effector is smoothed using the following algorithm: A spring model filtering algorithm is used to process the position data of the virtual robotic arm's end effector. The mathematical expression of the spring model is as follows: （1） （2） wherein, represents the target position of the real robot arm at this moment, represents the target position of the real robot arm at the last moment, represents the desired position of the real robot arm, i.e. the position output by the glasses, represents the tracking error, the parameter is the spring model stiffness coefficient, i.e. the proportional coefficient of the tracking error compensation; If parameter To ensure a constant value, when the tracking error is large, the actual robotic arm approaches the desired position faster, and the larger the tracking error, the faster the actual robotic arm moves. Simultaneously, to ensure smooth changes in the joint positions of the robotic arm during fixed-posture conditions and reduce angular velocity fluctuations, the design... It varies with the actual tracking error of the robotic arm; （3） Among them, parameters The amplitude of the sine function, parameter for The boundary value between two trends that change with tracking error, parameter The offset is a sine function, thus achieving the effect when the tracking error is greater than or equal to... At that time, the real robotic arm with The robot moves at a safe, constant speed towards the desired position; as the robotic arm moves, the tracking error decreases to [value missing]. Afterwards, the tracking speed of the robotic arm smoothly decreased; The virtual robotic arm's end-effector posture is represented by quaternions, and the posture data is processed using a spherical interpolation algorithm. The mathematical expression of spherical interpolation is: （4） Among them, parameters Represents the initial attitude unit quaternion, parameters Represents the final attitude unit quaternion, parameters express , The radian angle between two quaternions, parameters This represents the spherical interpolation scaling factor, which makes the virtual end-effector posture data received by the real robotic arm change smoothly. Step S4: During Cartesian space teaching, the glasses use the joint angle data output in step S3 to synchronously update and draw the virtual robotic arm, so that the virtual robotic arm produces the same movements as the real robotic arm, thereby completing the teaching.

2. The robotic arm teaching method based on mixed reality according to claim 1, characterized in that, In step S1, after the user puts on the mixed reality glasses, the mixed reality glasses perceive the surrounding environment and determine their own position through their built-in camera and sensors. Then, the virtual robotic arm and part of the virtual teaching scene are integrated into the mixed reality environment and presented to the user. At the same time, the glasses recognize the user's gestures, posture and voice information to realize the interaction between the user and the virtual robotic arm.

3. The robotic arm teaching method based on mixed reality according to claim 1, characterized in that, Step S2 includes the following steps: (1) Data composition The virtual robotic arm joint angle data includes angle information of six rotational joints and gripper opening and closing state information. The virtual robotic arm end tool's pose data relative to the base includes three-dimensional position information and three-dimensional posture information. In addition, the data also includes a flag to distinguish between the two types of data so that the controller in step S3 executes different programs. (2) Data conversion Because the coordinate system o-xyz of the mixed reality environment in the glasses is different from the coordinate system O-XYZ of the real environment in which the robotic arm is located, and because the communication method used has requirements on the data type, the data needs to be converted accordingly. (3) Joint angle initialization If step S1 is Cartesian space teaching, the real robotic arm initializes each joint and outputs the joint initialization angle to the glasses. After receiving the data, the glasses update and draw the virtual robotic arm. The virtual and real robotic arms overlap, thereby enhancing the sense of presence. Subsequently, the glasses continuously acquire and output the pose data of the virtual robotic arm end tool relative to the base, and wait for the real robotic arm controller to input data.

4. The robotic arm teaching method based on mixed reality according to claim 1, characterized in that, In step S4, during the Cartesian space teaching, the glasses use the joint angle data output by the real robotic arm controller in step S3 to synchronously update and draw the virtual robotic arm, so that the movement of the virtual robotic arm is the same as the movement of the real robotic arm, making the robotic arm movement of the mixed reality teaching closer to reality. At the same time, the mixed reality is used to know the expected position of the robotic arm end in the real environment in advance, avoiding collisions between the real robotic arm and the work scene, thereby completing the teaching.

Images