An augmented reality-based teaching assembly method for a robot arm

Abstract

The application relates to a kind of mechanical arm teaching assembly methods based on augmented reality, method includes: operator wears AR equipment and scans mechanical arm base QR code to complete virtual real registration;Obstacle data is collected using AR equipment depth sensor and is transmitted to upper computer ROS to construct environment model;During teaching process, ROS obtains joint data of mechanical arm in real time and drives virtual mechanical arm synchronous movement;At the same time, the shortest distance between mechanical arm and obstacle is calculated in real time based on the collision detection module of ROS MoveIt and FCL flexible collision library, and the distance is visualized in the form of red line and numerical value in AR field of view;When the distance is less than safety threshold, trigger early warning and pause physical mechanical arm movement, continue to execute assembly operation after operator confirms safety.The application upgrades the traditional "man-man" monitoring mode to "man-machine" intelligent cooperation mode through the risk early warning mechanism of virtual real integration, enhances the environmental risk perception ability of operator, reduces the collision probability, and improves the safety and efficiency of complex assembly task.

Description

Technical Field

[0001] This invention belongs to the field of industrial robot control and human-computer interaction technology, and specifically relates to a robotic arm teaching method based on Augmented Reality (AR). Background Technology

[0002] Robotic arms are widely used in the assembly of large and complex products such as spacecraft, large aircraft, and ships due to their high precision, high load capacity, and programmability. However, the internal structures of these products are typically extremely complex, with compact spaces filled with various precision equipment, cables, and pipes, forming a typical "constrained space." Online teaching of robotic arms within such spaces presents significant challenges:

[0003] 1. Blind spots and operational risks: Operators often cannot directly observe the end effector or the entire movement of the robotic arm, which can easily lead to collisions between the robotic arm and expensive surrounding equipment due to misoperation, resulting in significant economic losses.

[0004] 2. Inefficient due to reliance on manual monitoring: To ensure safety, traditional operation modes typically require "one operator and one monitor." The operator must constantly monitor the movement of the robotic arm and rely on the monitor's guidance to make experience-based judgments about the collision risks during assembly. This not only incurs high labor costs and cognitive load, but also suffers from subjectivity and lag in manual monitoring, making it impossible to guarantee safety and effectiveness.

[0005] 3. Traditional offline programming is not applicable: The constraint space environment is complex and ever-changing, and assembly tasks often have the characteristics of small batches and multiple varieties, making it difficult to generate perfect collision-free paths in advance through offline programming. Online teaching is still an indispensable link.

[0006] Augmented reality (AR) technology overlays virtual information onto the real world, providing operators with intuitive visual enhancements. Existing research has attempted to apply AR to industrial assembly, but most work focuses on simple scene annotation or virtual model overlay for cues, failing to deeply integrate real-time, high-precision collision detection capabilities and thus failing to form a complete, integrated "perception-decision-early warning" safety teaching solution.

[0007] Therefore, there is an urgent need for a new approach that can combine the intuitive display advantages of AR with real-time robot monitoring technology to solve the problem of safe teaching of robotic arms in constrained spaces. Summary of the Invention

[0008] The technical problem to be solved by this invention is to provide a robotic arm teaching assembly method based on augmented reality. By using AR technology to enhance the operator's ability to perceive potential risks, real-time collision detection, visualization and early warning mechanisms are deeply integrated into the teaching process to realize an intelligent mode of "one person operates, system monitors", which fundamentally reduces the risk of collision and improves the safety and efficiency of assembly operations.

[0009] The technical solution adopted in this invention is:

[0010] An augmented reality-based robotic arm teaching and assembly method includes the following steps:

[0011] S1: Virtual-Real Registration: Operators wear augmented reality devices and scan the markers fixed on the physical robotic arm base to complete the alignment and registration of the world coordinate systems of the virtual space and the real space.

[0012] S2: Environment Modeling: Using the depth sensor built into the augmented reality device, obstacle data in the robotic arm assembly work space is collected, and the data is transmitted to the host computer to construct a three-dimensional environment model for collision detection.

[0013] S3: Virtual-Real Linkage Teaching: The operator controls the movement of the physical robotic arm through the teaching pendant. The host computer acquires the joint data of the physical robotic arm in real time and drives the virtual robotic arm in the augmented reality device to move synchronously with it.

[0014] S4: Real-time collision detection and visualization: During the movement of the robotic arm, the host computer calculates the shortest distance d between the robotic arm model and the three-dimensional environment model in real time, and sends the shortest distance d and its spatial vector to the augmented reality device. In the augmented reality view, it is rendered as a visual line connecting the closest point between the robotic arm and the obstacle, and the value of the shortest distance d is displayed.

[0015] S5: Collision Warning and Decision Control: The host computer compares the shortest distance d with the preset safety threshold D. If d < D, the collision warning mode is triggered, a pause command is sent to the physical robotic arm controller, and a warning message is displayed in the augmented reality view. The operation can only continue after the operator confirms that it is safe.

[0016] S6: Repeat steps S3 to S5 until the robotic arm moves to the predetermined assembly point and completes the assembly operation.

[0017] Furthermore, in step S1, the identifier is a QR code, the QR code having its pose pre-defined in the physical robotic arm's base coordinate system; after the augmented reality device recognizes the QR code, it establishes the transformation relationship between the virtual model coordinate system and the physical robotic arm's base coordinate system through the following transformation matrix.

[0018]

[0019] In the formula, X virtual Y virtual Z virtual The position of the virtual robotic arm in its coordinate system is represented by X. real Y real Z real This represents the position of the physical robotic arm in its coordinate system. In the coefficient matrix, R and T represent the relative poses between the virtual and physical robots, respectively. R is the rotation matrix around the coordinate axis, and T is the three-dimensional translation vector.

[0020] Furthermore, in step S2, the host computer is a system running the Robot Operating System (ROS); in step S3, the host computer obtains the joint angle data of the physical robotic arm through the controller local area network bus, and drives the virtual robotic arm model in the augmented reality device to move synchronously through the TCP / IP communication protocol.

[0021] Furthermore, in step S4, during the robot arm's movement, the host computer, within the ROS operating system environment, calls the collision detection module in the MoveIt! motion planning framework to calculate the real-time distance between the robot arm's current motion state and the environmental model. First, the Bounding Volume Hierarchy (BVH) is used to quickly filter the robot arm link model and the environmental obstacle model. Bounding boxes are constructed to preliminarily determine potentially approaching object pairs, and the Euclidean distance between the centers of candidate bounding boxes is calculated, expressed as:

[0022]

[0023] Among them, (X) a ,Y a Z a (X) represents the center coordinates of the robotic arm link enclosure box. b ,Y b Z b ) represents the coordinates of the center of the bounding box of the environmental obstacle, and D represents the spatial distance between the centers of the two bounding boxes.

[0024] After completing the rapid bounding box screening, the system further calls the GJK (Gilbert–Johnson–Keerthi) algorithm from the flexible collision library FCL to accurately solve the distance between candidate geometries, calculating the shortest Euclidean distance d between the robotic arm link and environmental obstacles, which is mathematically expressed as:

[0025]

[0026] Among them, Link iDenotes the geometry of the i-th link of the robotic arm, Obstacle j Let represent the j-th obstacle geometry in the environment, and let GJK() function represent the shortest Euclidean distance between the two geometries. The algorithm calculates the shortest distance and returns the coordinates of the corresponding nearest points. The host computer generates a spatial vector connecting the two nearest points based on these coordinates and sends the shortest distance d and its spatial connection information to the augmented reality device. In the augmented reality view, this is rendered in real-time as a visual connection between the robotic arm and the obstacle, and the distance value is displayed simultaneously to indicate the operator's current safe distance between the robotic arm and the environment.

[0027] Furthermore, in step S4, the visualized connection is red, and the value of the shortest distance d is dynamically displayed next to the connection.

[0028] Furthermore, the security threshold D mentioned in step S5 is dynamically adjusted according to the specific application scenario.

[0029] Furthermore, the collision warning mode described in step S5 includes displaying a visual warning interface that says "Boundary distance is close!" overlaid in the augmented reality field of view.

[0030] Furthermore, the augmented reality device is the Microsoft HoloLens 2.

[0031] The positive effects of this invention are:

[0032] Compared with the prior art, the present invention has the following significant advantages:

[0033] (1) Enhance perception and eliminate blind spots: AR transforms the intangible collision distance information into tangible, always visible visual elements (red lines and numbers), which greatly enhances the operator's perception of spatial relationships and situational awareness, and makes up for the defects of blind spots.

[0034] (2) Real-time accuracy and proactive protection: The collision detection module of ROS MoveIt! and the FCL flexible collision library are used to achieve high-frequency and high-precision real-time collision detection. It can provide early warning before physical collisions occur, transforming passive "manual monitoring and assistance" into proactive "system monitoring and assistance", thus fundamentally avoiding collision accidents.

[0035] (3) Human-machine collaboration and safe decision-making: The system is responsible for providing accurate data and early warnings, while the final decision-making power is still given to the operator (confirmation is required before continuing), realizing the "human-in-the-loop" operation mode, which ensures safety while preserving human flexibility and judgment.

[0036] (4) Liberate manpower and improve efficiency: The traditional multi-person mode is transformed into a single-person mode, which saves valuable human resources, reduces labor costs, and reduces errors caused by poor communication and human fatigue, thus improving overall work efficiency. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the overall architecture of the present invention.

[0038] Figure 2 This is a schematic diagram illustrating the principle of the virtual-real mapping synchronization control of the present invention;

[0039] Figure 3 This is a detailed flowchart of the assembly method of the present invention. Detailed Implementation

[0040] As attached Figure 1-3 As shown, this invention discloses an augmented reality-based robotic arm teaching and assembly method, aiming to solve the problems of high collision risk and low efficiency caused by the need for dedicated personnel to supervise the teaching and assembly of large and complex products (such as spacecraft) in constrained spaces. Specifically, it includes the following steps:

[0041] S1: Environmental Perception and Virtual-Real Registration

[0042] Operators wear mixed reality headsets such as the Microsoft HoloLens 2. A pre-calibrated QR code is fixed to the base of the physical robotic arm. After the device is activated, it scans the QR code using its front-facing camera. Since the pose of the QR code in the robotic arm's base coordinate system is known, the system uses computer vision algorithms to calculate the transformation relationship between the AR virtual model's coordinate system and the physical robotic arm's base coordinate system, achieving virtual-real space registration and ensuring that the virtual model can be accurately superimposed on the real robotic arm and move accordingly. The QR code is fixed to the physical robotic arm's base, and the virtual-real registration coordinate transformation is achieved through the following transformation matrix:

[0043]

[0044] In the formula, X virtual Y virtual Z virtual The position of the virtual robotic arm in its coordinate system is represented by X. real Y real Z real This represents the position of the physical robotic arm in its coordinate system. In the coefficient matrix, R and T represent the relative poses between the virtual and physical robots, respectively. R is the rotation matrix about the coordinate axes, and T is the three-dimensional translation vector.

[0045] S2: Environment Modeling

[0046] After registration is complete, the operator uses the HoloLens 2's built-in depth sensor (Azure Kinect) to scan the work area and collect 3D point cloud data of obstacles in the environment (such as product parts, tooling, frames, etc.). This data is transmitted to the host computer in real time via Wi-Fi, where it is filtered, denoised, and segmented before being imported into the Robot Operating System (ROS) for coordinate system registration.

[0047] S3: Virtual and Real Interconnected Teaching

[0048] The operator uses a traditional physical teach pendant to control the movement of a real physical robotic arm. The control unit ROS communicates with both the physical robotic arm controller and the HoloLens 2 augmented reality device via TCP / IP. The physical robotic arm controller sends joint angle data streams to the ROS system in real time. A custom node in ROS subscribes to this data and forwards it to the HoloLens 2. Upon receiving the data, the application on HoloLens 2 drives the virtual robotic arm model in its scene to perform perfectly synchronized movements, thus achieving synchronous movement between the physical and virtual robotic arms and providing the operator with accurate visual feedback that links the virtual and real worlds.

[0049] S4: Real-time collision detection and visualization

[0050] Throughout the robotic arm's movement, the ROS system calls the collision detection module within the MoveIt! motion planning framework in the background to perform real-time distance calculations between the robotic arm's operating state and the surrounding working environment. First, it uses a Bounding Volume Hierarchy (BVH) to quickly filter the robotic arm link model and the environmental obstacle model. By constructing bounding boxes, it makes a preliminary judgment on potentially approaching object pairs and calculates the Euclidean distance between the centers of candidate bounding boxes, expressed as:

[0051]

[0052] Among them, (X) a ,Y a Z a (X) represents the center coordinates of the robotic arm link enclosure box. b ,Y b Z b ) represents the coordinates of the center of the bounding box of the environmental obstacle, and D represents the spatial distance between the centers of the two bounding boxes.

[0053] After completing the rapid bounding box selection, the system further calls the Flexible Collision Library (FCL) for real-time and accurate distance calculation. The core of the FCL library is the efficient GJK (Gilbert–Johnson–Keerthi) algorithm.

[0054] Its working principle is as follows: FCL constructs a convex hull representation for each link of the virtual robotic arm and each obstacle in the scene. For any pair of links... and obstacles The GJK algorithm calculates the closest point and the shortest Euclidean distance *d* between two convex shapes using an extremely efficient iterative method. The mathematical expression of this process is:

[0055]

[0056] Among them, Link i Denotes the geometry of the i-th link of the robotic arm, Obstacle j Let J represent the j-th obstacle geometry in the environment, and let GJK() function represent the shortest Euclidean distance function between two geometries.

[0057] The calculated shortest distance d and its corresponding spatial vector (i.e., the direction of the line connecting the nearest point on the robotic arm link and the nearest point on the obstacle, determined by the GJK algorithm) are sent back to HoloLens in real time. In the AR field of view, HoloLens renders this vector line as a consistently red line and dynamically displays the distance value (e.g., "0.12m") in real-time next to it. This "red line" visually indicates the location and relative position of the nearest obstacle, providing the operator with crucial spatial awareness information and intuitively revealing potential collision risks.

[0058] S5: Collision Warning and Decision Control

[0059] The ROS system will continuously monitor the shortest distance d. The system has a preset safety threshold D (adjustable according to mission requirements, e.g., 0.1 meters). Once d < D is detected, the system immediately triggers a collision warning mode.

[0060] (1) Decision control: ROS sends an emergency stop command to the controller of the physical robotic arm to stop all movement.

[0061] (2) Visual warning: In the AR field of view of HoloLens, a prominent "close to the boundary" warning interface will pop up, obscuring part of the field of view and forcing the operator to pay attention.

[0062] (3) Manual intervention: The operator must visually confirm the situation on site. After confirming safety, the operator should operate the teach pendant again to move away from the danger zone before the warning interface will be cleared.

[0063] S6: Loop execution and assembly completion

[0064] The operator repeats steps S3 to S5, gradually and carefully guiding the robotic arm towards the target point. The system continuously monitors the process until the end effector of the robotic arm safely and without collision reaches the final assembly point, completing the assembly task.

[0065] The specific steps for using this invention are as follows:

[0066] 1. Preparation: The operator arrives at the assembly site, obtains the assembly points, and wears the HoloLens 2 device.

[0067] 2. Add virtual models: Operators launch applications in HoloLens 2 and add virtual machines to the real scene.

[0068] 3. Virtual-to-real registration: Registration is completed by scanning a QR code using a HoloLens 2 device. The holographic image of the virtual robotic arm is superimposed on the physical robotic arm.

[0069] 4. Environmental scanning: Operators walk around the work area and use HoloLens 2 to scan the obstacle information of the assembly site. The 3D environmental data is processed and loaded into ROS to achieve real-time mapping with the real environment.

[0070] 5. Teaching Control: The operator uses a teach pendant to slowly move the robotic arm to the assembly point. In the AR view, the virtual robotic arm moves in real time.

[0071] 6. Real-time monitoring: When the robotic arm moves, a red line automatically appears connecting the robotic arm to the nearest obstacle, indicating the closest point between the robotic arm and the obstacle, with "0.12m" displayed next to it. Operators can clearly perceive the location of risks.

[0072] 7. Collision Warning Triggered: The operator continues fine-tuning, reducing the distance to 0.08m (less than the preset threshold of 0.1m). The robotic arm immediately and automatically pauses its movement, and a red warning pops up in the center of the AR field of view: "Boundary distance close!" The operator stops operation and observes to confirm that there is still a gap between the robotic arm and the obstacle, and there is no risk of collision.

[0073] 8. Collision warning canceled: The operator moves the robotic arm away from the assembly again, the warning interface disappears, and the operator selects a path slightly away from the obstacle to continue moving.

[0074] 9. Assembly Completion: Under the full visual monitoring of the system, the robotic arm safely and accurately arrives at the assembly point, where the operator completes the assembly operation. The entire process is completed independently by only one person, ensuring safety and efficiency.

[0075] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robotic arm teaching and assembly method based on augmented reality, characterized in that... It includes the following steps: S1: Virtual-Real Registration: Operators wear augmented reality devices and scan the artificial markers fixed on the physical robotic arm base to complete the alignment and registration of the world coordinate systems of the virtual space and the real space. S2: Environment Modeling: Using the depth sensor built into the augmented reality device, obstacle data in the workspace of the robotic arm is collected and transmitted to the host computer to construct a three-dimensional environment model for collision detection. S3: Virtual-Real Linkage Teaching: The operator controls the movement of the physical robotic arm through the teaching pendant. The host computer acquires the joint data of the physical robotic arm in real time and drives the virtual robotic arm in the augmented reality device to move synchronously with it. S4: Real-time collision detection and visualization: During the movement of the robotic arm, the host computer calculates the shortest distance d between the robotic arm model and the three-dimensional environment model in real time, and sends the shortest distance d and its spatial vector to the augmented reality device. In the augmented reality view, it is rendered as a visual line connecting the closest point between the robotic arm and the obstacle, and the value of the shortest distance d is displayed. S5: Collision Warning and Decision Control: The host computer compares the shortest distance d with the preset safety threshold D. If d < D, the collision warning mode is triggered, a pause command is sent to the physical robotic arm controller, and a warning message is displayed in the augmented reality view. The operation can only continue after the operator confirms that it is safe. S6: Repeat steps S3 to S5 until the robotic arm moves to the predetermined assembly point and completes the assembly operation.

2. The augmented reality-based robotic arm teaching and assembly method according to claim 1, characterized in that... The identifier mentioned in step S1 is a QR code, whose pose in the physical robotic arm's base coordinate system is pre-defined. Using the camera system built into the augmented reality device, after recognizing the QR code based on computer vision, the transformation relationship between the virtual model coordinate system and the physical robotic arm's base coordinate system is established through the following transformation matrix: In the formula, X virtual Y virtual Z virtual The position of the virtual robotic arm in its coordinate system is represented by X. real Y real Z real This represents the position of the physical robotic arm in its coordinate system. In the coefficient matrix, R and T represent the relative poses between the virtual and physical robots, respectively. R is the rotation matrix around the coordinate axis, and T is the three-dimensional translation vector.

3. The augmented reality-based robotic arm teaching and assembly method according to claim 1, characterized in that... In step S2, the host computer is a system running the Robot Operating System (ROS). In step S3, the host computer obtains the joint angle data of the physical robotic arm through the controller local area network bus and drives the virtual robotic arm model in the augmented reality device to move synchronously through the TCP / IP communication protocol.

4. The augmented reality-based robotic arm teaching and assembly method according to claim 1, characterized in that... In step S4, real-time collision detection occurs during the robot arm's movement. The host computer, operating under the ROS (Robot Operating System) environment, calls the collision detection module within the MoveIt! motion planning framework to calculate the real-time distance between the robot arm's current motion state and the environmental model. First, a bounding box hierarchy is used to quickly filter the robot arm link model and the environmental obstacle model. Bounding boxes are constructed to preliminarily determine potentially close object pairs, and the Euclidean distance between the centers of candidate bounding boxes is calculated. The expression for this distance is: Among them, (X) a ,Y a Z a (X) represents the center coordinates of the robotic arm link enclosure box. b ,Y b Z b ) represents the coordinates of the center of the bounding box of the environmental obstacle, and D represents the spatial distance between the centers of the two bounding boxes. After completing the rapid bounding box screening, the system further calls the GJK algorithm in the Flexible Collision Library (FCL) to accurately solve the distance between candidate geometries, calculating the shortest Euclidean distance d between the robotic arm link and environmental obstacles, which is mathematically expressed as: Among them, Link i Denotes the geometry of the i-th link of the robotic arm, Obstacle j Let represent the j-th obstacle geometry in the environment, and let GJK() function represent the shortest Euclidean distance between the two geometries. The algorithm calculates the shortest distance and returns the coordinates of the corresponding nearest points. The host computer generates a spatial vector connecting the two nearest points based on these coordinates and sends the shortest distance d and its spatial connection information to the augmented reality device. In the augmented reality view, this is rendered in real-time as a visual connection between the robotic arm and the obstacle, and the distance value is displayed simultaneously to indicate the operator's current safe distance between the robotic arm and the environment.

5. A robotic arm teaching and assembly method based on augmented reality according to claim 1 or 4, characterized in that... In step S4, the visual connection is displayed in red, and the value of the shortest distance d is dynamically displayed next to the connection.

6. The augmented reality-based robotic arm teaching and assembly method according to claim 1, characterized in that... The safety threshold D mentioned in step S5 is dynamically adjusted according to the specific assembly application scenario.

7. The augmented reality-based robotic arm teaching and assembly method according to claim 1, characterized in that... The collision warning modality described in step S5 includes displaying a visual warning interface that overlays "Boundary distance is close!" in the augmented reality field of view.

8. The augmented reality-based robotic arm teaching and assembly method according to claim 1, characterized in that... The augmented reality device mentioned is the Microsoft HoloLens 2.

Images