A remote laser repair method based on virtual fixture and autonomous agent

Abstract

The application discloses a remote laser repair method based on a virtual clamp and an autonomous agent and belongs to the technical field of robot teleoperation. The virtual clamp is designed to provide force guidance and motion constraint for an operator; secondly, a normal vector of a repair point is determined according to point cloud data of a workpiece to be repaired, and an optimal repair angle and height are predicted according to a molten pool state and laser power; finally, an autonomous agent is used to automatically determine joint angles of a mechanical arm to keep the repair angle and height. The application reduces the operation difficulty of remote repair, shortens the training time and improves the effectiveness of remote operation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of robot teleoperation, and particularly relates to a remote laser repair method based on virtual fixtures and autonomous agents. BACKGROUND

[0002] In modern industrial production, although laser repair robots are superior to human welders in terms of precision, stability and response to dangerous environments, when facing unpredictable factors such as workpiece variation and poor assembly in actual production, the pre-programmed laser repair scheme of the robot often cannot cope with them. Therefore, they can usually only work effectively in a highly preset and structured environment. In contrast, skilled human welders adaptively adjust laser repair operations through their perception, analysis and information synthesis capabilities, especially in critical laser repair tasks and unstructured work environments. However, high-risk work environments and long-term laser repair operations also pose health risks to human welders, such as inhaling harmful smoke, gas and arc radiation.

[0003] At present, the robot remote laser repair strategy with human-in-the-loop has become a feasible solution, which can enable human welders to perform laser repair operations away from these dangerous and harmful environments, while effectively combining the advantages of humans and robots, with higher motion precision and adaptive adjustment capabilities.

[0004] In remote operation tasks, direct control often requires a high cognitive workload while maintaining high concentration. Therefore, operators must undergo extensive training. In addition, in laser repair tasks, operators must overcome the difficulty of maintaining a fixed angle between the welding gun and the workpiece and adapting to the height along the laser repair path. Therefore, if the remote operator wants to obtain good laser repair results, 6 degrees of freedom control is required, which undoubtedly places a heavy burden on the operator. SUMMARY

[0005] To solve the above problems, the present application discloses a remote laser repair method based on virtual fixtures and autonomous agents, which introduces virtual fixtures with motion constraints and force guidance to shorten the training time and thus improve the effectiveness of remote operation, while using a shared control strategy and using autonomous agents to automatically maintain the angle and height of welding, so that the operator only needs to perform 2 degrees of freedom control, thereby reducing the mental burden of the operator and improving the task execution ability.

[0006] To achieve the above purpose, the technical scheme of the present application is as follows:

[0007] The master system of the scheme is developed based on Unity3D, which is used for reconstructing the workpiece to be repaired and selecting the laser repair path; the slave system is developed based on ROS, which is connected with the mechanical arm, laser welding gun, infrared camera and depth camera, wherein the laser welding gun, infrared camera and depth camera are rigidly connected with the end of the mechanical arm; the master system and the slave system realize data transmission through the 5G network.

[0008] A remote laser repair method based on virtual fixture and autonomous agent, comprising the following steps:

[0009] 1) Reference direction and error direction of welding gun movement:

[0010] The reference laser repair path selected by the operator is described by a parametric expression:

[0011] p(s)≡(x(s),y(s),z(s)) T ,s∈[a,b]

[0012] Where s is the parameter, and [a, b] is the change range of s. The current Cartesian position m c of the master virtual welding gun is:

[0013] m c =(x c ,y c ,z c ) T

[0014] When the master virtual welding gun is not accurately located on the reference path, the reference direction δ p (m c ) and the error direction e(m c ) of the point are defined as:

[0015]

[0016] Where d represents the instantaneous movement direction vector along the path, is the nearest point on the reference path from the current position, is the normalized tangent vector at the point.

[0017] 2) Master force guidance and movement constraint:

[0018] When the virtual welding gun movement deviates from the reference path, the virtual fixture gives the operator force feedback to correct this error; at the same time, the operator is given a driving force in the direction of change along the reference path, and finally the operator is given a resultant force F c :

[0019]

[0020] Where F maxF is the maximum value of the error feedback force p k is the driving force along the path e k p is the adjustment factor. In order to make the real repair path close to the reference path, the virtual welding gun position m w is defined with constraints

[0021] m w = m c -k s e(m c )

[0022] where k s is an adjustment factor positively correlated with k e . When k s = 1, it is a hard constraint, at which the laser repair position is completely constrained, and the operator can only advance or retreat along the reference path. When 0 < k s < 1, it is a soft constraint, at which the laser repair position will be close to the reference path, but the operator can also deviate from the reference path under force feedback.

[0023] 3) From the end real repair position and normal vector:

[0024] In order to ensure that the repair point is actually on the workpiece surface, we use the projection of the virtual welding gun position on the workpiece surface as the real laser repair point m r , denoted as:

[0025] m r = Raycast(m w )

[0026] At the same time, the normal vector n of the real laser repair point is calculated according to the point cloud data around the point.

[0027] 4) Determine the welding gun inclination and height:

[0028] According to the image of the molten pool during laser repair and the laser power, the CNN-LSTM network is used to predict the optimal angle θ and height h of the welding gun.

[0029] 5) Autonomous agent determines the end of the mechanical arm pose:

[0030] Taking {δ p (m c ), n x δ p (m c ), n} as the local coordinate system of the laser repair point m r , the pose T weld of the point is obtained. The optimal inclination and distance θ, h of the laser repair correspond to the change matrix T θ,h , so the pose T robot of the end of the mechanical arm is determined.:

[0031] T robot =T weld T θ,h T tool -1

[0032] where T tool is the transformation matrix from the end of the robot arm to the end of the welding torch.

[0033] Further, the calculation method of the real repair point normal vector in step 3) is:

[0034] Based on the least square method, the m r points are fitted to a three-dimensional plane, and the neighborhood points are marked as P i , then the distance from the point to the plane can be defined as d i = (P i -A) · n, A represents the centroid of the points in the neighborhood range, and is defined as:

[0035]

[0036] When d i = 0 is obtained by the least square method, n is the surface normal vector, and the surface normal vector n is calculated by principal component analysis. First, a covariance matrix C is constructed:

[0037]

[0038] By solving the eigenvalue λ and eigenvector V of C, the surface normal vector n is found:

[0039]

[0040] where ζ i is the weight value of P i point, and the contribution of all neighborhood points to the normal vector estimation is consistent, and ζ i = 1 is set. Since C is symmetric and semi-positive definite, its eigenvalue λ is real. If the eigenvalue corresponding to the minimum eigenvector is greater than 0, then the eigenvector is the estimated value of the surface normal vector, n = {n x , n y , n z}.

[0041] The beneficial effects of the method are:

[0042] The method uses a virtual clamp with motion constraints and force guidance to help operators move the remote tool along the preferred path through enhanced guidance force, improve remote operation performance, reduce total workload, and ensure safety during task execution; autonomous agents are used to automatically maintain the angle and height of the welding, so that the operator only needs to control 2 degrees of freedom, thereby reducing the mental burden of the operator and improving task execution. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The system block diagram of the embodiment of the present application.

[0044] Figure 2 The main flowchart of the embodiment of the present application. DETAILED DESCRIPTION

[0045] The present application will be further illustrated in conjunction with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0046] As Figure 1 shown, a remote laser repair method based on a virtual clamp and an automatic agent,

[0047] The master system is developed based on Unity3D, connecting VR devices and hand controllers, and users view the reconstructed three-dimensional workpiece through the VR device and select the laser repair path using the hand controller. The slave system is developed based on ROS, connecting the mechanical arm, laser welding gun, infrared camera, and depth camera. The depth camera is used to collect 3D point cloud data of the workpiece to be repaired for reconstruction in the master system; the infrared camera is used to collect the molten pool image of the laser repair to determine the current state; the connection between the mechanical arm and the laser welding gun is the main execution device for laser repair, which is used to perform real laser repair according to the mechanical arm pose and laser control information transmitted by the master.

[0048] Figure 2 The specific method of the embodiment is shown, which includes the following steps:

[0049] Step one, determine the reference direction and error direction of motion according to the virtual welding gun position input by the user:

[0050] Suppose the reference laser repair path selected by the operator is described by a parametric expression:

[0051] p(s)≡(x(s),y(s),z(s)) T ,s∈[a,b]

[0052] Where s is the parameter, and [a, b] is the range of s. The current Cartesian position m c of the master virtual welding gun is:

[0053] m c = (x c ,y c ,z c ) T

[0054] When the master virtual welding torch is not precisely on the reference path, the point on the reference path closest to the current position can be expressed as:

[0055]

[0056] Thus the reference direction δ p (m c ) and the error direction e(m c ) of the current point are defined as:

[0057]

[0058] where d represents the instantaneous motion direction vector along the path, and n(m ) is the normalized tangent vector at the point.

[0059] Step two, determine the size of the master guidance force and motion constraints:

[0060] When the virtual welding torch moves away from the reference path, the virtual fixture gives the operator force feedback to correct this error; at the same time, it gives the operator a driving force in the direction of change along the reference path, and finally gives the operator a resultant force F c :

[0061]

[0062] where F max is the maximum value of the error feedback force, F p is the driving force size along the path motion, k e , k p are adjustment factors. In order to make the real repair path close to the reference path, define the virtual welding torch position m w with constraints as the actual input position:

[0063] m w = m c -k s e(m c )

[0064] where k s is an adjustment factor that is positively correlated with k e . When k s = 1, it is a hard constraint, at this time the laser repair position is completely constrained, the operator can only advance or retreat along the reference path. When 0 < k sWhen <1, it is soft constraint, the laser repair position will be close to the reference path, but the operator can also deviate from the reference path under force feedback.

[0065] Step three, determine the normal vector of the real repair position from the point:

[0066] In order to ensure that the welding point is really on the workpiece surface, we use the projection of the virtual welding gun position on the workpiece surface as the real welding point m r This function uses the Raycast function of unity3d, recorded as:

[0067] m r =Raycast(m w )

[0068] The normal vector n of the point is obtained by the following method:

[0069] Based on the least square method, the three-dimensional plane where m r is fitted, and the neighborhood points are marked by P i , then the distance from the point to the plane can be defined as d i =(P i -A)·n, A represents the centroid of the points in the neighborhood range, which is defined as:

[0070]

[0071] After the least square method, n when d i =0 is obtained, which is the surface normal vector. The surface normal vector n is calculated by principal component analysis. First, construct the covariance matrix C:

[0072]

[0073] By solving the eigenvalue λ and eigenvector V of C, the surface normal vector n is found:

[0074]

[0075] Where, ζ i is the weight value of P i point, in this paper, the contribution of all neighborhood points to the estimation of normal vector is consistent, set ζ i =1. Since C is symmetric and semi-positive definite, its eigenvalue λ is real. If the minimum eigenvalue corresponds to the eigenvalue greater than 0, then the eigenvalue is the estimated value of the surface normal vector, n={n x ,n y ,n z}.

[0076] Step four, determine the repair angle and height according to the current molten pool state and laser power:

[0077] The image sequence of the melt pool is represented as {I1,I2,...,I T}, where each image I t The size of is H×W×C; the laser power sequence is represented as {P1,P2,...,P T}.

[0078] For each time step t, we use CNN to extract the image I t Extract features:

[0079] f t =CNN(I t )

[0080] The feature vector f extracted by CNN t With laser power P t Combine to form a merged feature vector:

[0081] z t =[f t ,P t ]

[0082] The merged feature sequence {z1,z2,...,z T} Input LSTM network for time series processing:

[0083] h t =LSTM(z t ,h t-1 )

[0084] The hidden state of the LSTM is converted into the angle and height of the welding gun through the fully connected layer:

[0085] y t =[θ t ,h t ]=W y h t +b y

[0086] where y t is the output vector, W y is the weight matrix, b y is the bias vector.

[0087] Step 5: Use the autonomous agent to automatically determine the position of the slave robot arm:

[0088] With {δ p (m c ),n×δ p (m c ),n} as the laser repair point m r The local coordinate system of the point is obtained by weldThe optimal inclination angle and distance of laser repair θ, h, and the corresponding change matrix are T θ,h The pose of the end of the mechanical arm T robot :

[0089] T robot = T weld T θ,h T tool -1

[0090] Wherein T tool is the transformation matrix from the end of the mechanical arm to the end of the welding torch. According to the pose of the end of the mechanical arm, the corresponding joint angle J virtual is solved by inverse kinematics, and these angles are sent to the mechanical arm controller to achieve the required repair motion.

[0091] The embodiment uses a virtual clamp with motion constraints and force guidance to help the operator move the remote tool along the preferred path through enhanced guidance force, improve remote operation performance, reduce the total workload, and ensure the safety of task execution; at the same time, an autonomous agent is used to automatically maintain the angle and height of welding, so that the operator only needs to control two degrees of freedom, thereby reducing the mental burden of the operator and improving the task execution force.

[0092] It should be noted that the above content only illustrates the technical idea of the present application, and cannot be used to limit the protection scope of the present application. For ordinary skilled persons in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which fall within the scope of protection of the claims of the present application.

Claims

1. A remote laser repair method based on virtual fixture and autonomous agent, characterized in that: The following steps are involved: 1) Reference direction and error direction of welding gun movement: The reference laser repair path selected by the operator is described by the parameter expression: ; in is a parameter, yes The current Cartesian position of the virtual welding gun at the master end for: ; Define the reference direction of the master virtual welding gun when it is not exactly on the reference path and error direction for: ; ; ; in represents the instantaneous motion direction vector along the path, is the point on the reference path closest to the current position, is the normalized tangent vector at the point; 2) Master end force guidance and motion constraints: When the virtual welding gun deviates from the reference path, the virtual fixture gives the operator force feedback to correct the error; at the same time, it gives the operator a driving force along the direction of the reference path, and finally gives the operator a resultant force. : ; in is the maximum value of the error feedback force, is the driving force along the path, 、 is the regulating factor; In order to make the real repair path close to the reference path, define the virtual welding gun position with constraints As actual input location: ; in is with A positively correlated moderating factor; =1 is a hard constraint, at which point the laser repair position is completely constrained and the operator can only move forward or backward along the reference path; when 0< When <1, it is a soft constraint. In this case, the laser repair position will be close to the reference path, but the operator can also deviate from the reference path under force feedback. 3) Repair the real position and normal vector from the client: In order to ensure that the repair point actually falls on the workpiece surface, the projection of the virtual welding gun position on the workpiece surface is used as the real laser repair point. , recorded as: ; At the same time, the normal vector of the point is calculated based on the point cloud data around the real laser repair point ; 4) Determine the welding gun inclination angle and height: The CNN-LSTM network is used to predict the optimal angle of the welding gun based on the image of the molten pool and the laser power during laser repair. and height ; 5) The autonomous agent determines the position of the slave robot arm: by As a laser repair point The local coordinate system of the point is obtained ;Optimal inclination angle and distance for laser repair 、 , the corresponding change matrix is , then the posture of the end of the robot arm : ; in is the transformation matrix from the end of the robot arm to the end of the welding gun.

2. The remote laser repair method based on virtual fixture and autonomous agent according to claim 1 is characterized in that: It includes a master system and a slave system. The master system is developed based on Unity3D and is used to reconstruct the workpiece to be repaired and select the laser repair path; the slave system is developed based on ROS and connects the robotic arm, laser welding gun, infrared camera and depth camera, among which the laser welding gun, infrared camera and depth camera maintain a rigid connection with the end of the robotic arm; the master system and the slave system realize data transmission through the 5G network.

3. The remote laser repair method based on virtual fixture and autonomous agent according to claim 1 is characterized in that: Step 3) The normal vector calculation method is: Least squares fitting The three-dimensional plane where the neighboring points are located is Mark, then the distance from the point to the plane can be defined as , A represents the centroid of the points within the neighborhood, which is defined as: ; By the least squares method, we get Time , which is the surface normal vector, the surface normal vector It is calculated using principal component analysis. First, the covariance matrix C is constructed: ; By solving the eigenvalue λ and eigenvector V of C, we can find the surface normal vector : ; in, P i The weight value of the point, all neighboring points contribute the same amount to the normal vector estimation, set =1; Since C is symmetric and semi-positive, its eigenvalues ​​λ are all real numbers. If the eigenvalue corresponding to the minimum eigenvector is greater than 0, then the eigenvector is the estimated value of the surface normal vector. .

4. The remote laser repair method based on virtual fixture and autonomous agent according to claim 1 is characterized in that: In step 4), the optimal angle and height of the welding gun are determined according to the current molten pool state and laser power: The image sequence of the melt pool is represented as , where each image The size is ; The laser power sequence is expressed as ; For each time step , using CNN from images Extract features: ; Will Extracted feature vectors With laser power Combine to form a merged feature vector: ; Merge feature sequences Input LSTM network for time series processing: ; The hidden state of the LSTM is converted into the angle and height of the welding gun through the fully connected layer: ， in is the output vector, is the weight matrix, is the bias vector.

Images