Remote laser repair method based on virtual fixture and autonomous agent

Abstract

The invention discloses a remote laser repair method based on a virtual fixture and an autonomous agent, and belongs to the technical field of robot teleoperation. According to the method, the virtual clamp is designed to provide force guidance and motion constraint for an operator; secondly, determining a normal vector of a repair point according to the point cloud data of the to-be-repaired workpiece, and predicting an optimal repair angle and height according to a molten pool state and laser power; and finally, automatically determining the joint angle of the mechanical arm by using an autonomous agent so as to maintain the repair angle and height. According to the invention, the operation difficulty of remote repair is reduced, the training time is shortened, and the effectiveness of remote operation is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of robot teleoperation, and in particular relates to a remote laser repair method based on a virtual fixture and an autonomous agent. Background Art

[0002] In modern industrial production, although laser repair robots are superior to human welders in terms of accuracy, stability and handling of hazardous environments, the robots' pre-programmed laser repair solutions are often unable to cope with unpredictable factors common in actual production, such as workpiece variation and poor assembly. Therefore, they can usually only work effectively in highly preset and structured environments. In contrast, skilled human welders use their perception, analysis and information synthesis capabilities to adaptively adjust laser repair operations, especially in critical laser repair tasks and unstructured working environments. However, high-risk working environments and long laser repair operations expose human welders to health risks, such as inhalation of harmful fumes, gases and arc radiation.

[0003] Currently, the human-in-the-loop robotic remote laser repair strategy has become a viable solution that allows human welders to escape these dangerous and harmful environments when performing laser repair operations, while effectively combining the advantages of humans and robots with higher motion accuracy and adaptive adjustment capabilities.

[0004] In remote operation tasks, direct control usually requires a high cognitive workload and requires a high level of concentration. Therefore, operators must receive extensive training. In addition, in laser repair tasks, operators must overcome the difficulty of maintaining a fixed angle and adapting to the height between the welding gun and the workpiece while operating along the laser repair path. Therefore, if remote operators want to achieve good laser repair results, they need to control 6 degrees of freedom, which undoubtedly brings a huge burden to the operators. Summary of the invention

[0005] To solve the above problems, the present invention discloses a remote laser repair method based on a virtual fixture and an autonomous agent. A virtual fixture with motion constraints and force guidance is introduced to shorten the training time and thus improve the effectiveness of remote operation. At the same time, a shared control strategy is used and an autonomous agent is adopted to automatically maintain the angle and height of the welding, so that the operator only needs to perform control of 2 degrees of freedom, thereby reducing the operator's mental burden and improving task execution.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] The master system of this solution is developed based on Unity3D, which is used to reconstruct the workpiece to be repaired and select the laser repair path; the slave system is developed based on ROS, connecting the robotic arm, laser welding gun, infrared camera and depth camera, among which the laser welding gun, infrared camera and depth camera are rigidly connected to the end of the robotic arm; the master system and the slave system realize data transmission through the 5G network.

[0008] A remote laser repair method based on a virtual fixture and an autonomous agent comprises the following steps:

[0009] 1) Reference direction and error direction of welding gun movement:

[0010] The reference laser repair path selected by the operator is described by the parameter expression:

[0011] p(s)≡(x(s),y(s),z(s)) T ,s∈[a,b]

[0012] Where s is a parameter and [a,b] is the range of s. The current Cartesian position of the virtual welding gun at the master end is m c for:

[0013] m c =(x c ,y c ,z c ) T

[0014] When the master end virtual welding gun is not exactly located on the reference path, the reference direction δ of the point is defined p (m c ) and error direction e(m c )for:

[0015]

[0016] Where d represents the instantaneous motion direction vector along the path, is the point on the reference path closest to the current position, is the normalized tangent vector at this point.

[0017] 2) Main end force guidance and motion constraint:

[0018] When the virtual welding gun deviates from the reference path, the virtual fixture gives the operator force feedback to correct the error; at the same time, it gives the operator a driving force along the direction of the reference path, and finally gives the operator a resultant force F c :

[0019]

[0020] where F maxis the maximum value of the error feedback force, F p is the driving force along the path, k e , k p In order to make the real repair path close to the reference path, the virtual welding gun position m with constraints is defined. w As actual input location:

[0021] m w =m c -k s e(m c )

[0022] where k s is with k e A positively correlated regulatory factor. s = 1 is a hard constraint, in which case the laser repair position is completely constrained and the operator can only move forward or backward along the reference path. <k s When <1, it is a soft constraint. At this time, the laser repair position will be close to the reference path, but the operator can also deviate from the reference path under force feedback.

[0023] 3) The actual repair position and normal vector from the end:

[0024] In order to ensure that the repair point actually falls on the workpiece surface, we use the projection of the virtual welding gun position on the workpiece surface as the real laser repair point m r , recorded as:

[0025] m r =Raycast(m w )

[0026] At the same time, the normal vector n of the point is calculated based on the point cloud data around the real laser repair point.

[0027] 4) Determine the welding gun inclination angle and height:

[0028] According to the image of the molten pool and the laser power during laser repair, the CNN-LSTM network is used to predict the optimal angle θ and height h of the welding gun.

[0029] 5) The autonomous agent determines the position of the slave robot arm:

[0030] With {δ p (m c ),n×δ p (m c ),n} as the laser repair point m r The local coordinate system of the point is obtained by weld The optimal inclination angle and distance θ, h for laser repair, and the corresponding change matrix is ​​T θ,h , then the position of the end of the robot arm is T robot:

[0031] T robot =T weld T θ,h T tool -1

[0032] Where T tool is the transformation matrix from the end of the robot arm to the end of the welding gun.

[0033] Furthermore, the calculation method for calculating the normal vector of the real repair point in step 3) is:

[0034] Fitting m based on the least squares method r The three-dimensional plane where the neighboring points pass through P i Mark, then the distance from the point to the plane can be defined as d i =(P i -A)·n, A represents the centroid of the points in the neighborhood, which is defined as:

[0035]

[0036] The least squares method is used to obtain d i = 0, that is, the surface normal vector n, which is calculated using principal component analysis. First, the covariance matrix C is constructed:

[0037]

[0038] By solving the eigenvalue λ and eigenvector V of C, we can find the surface normal vector n:

[0039]

[0040] Among them, i P i The weight value of the point, in this paper, all neighborhood points contribute the same amount to the normal vector estimation, and set ζ i =1. Since C is symmetric and semi-positive definite, its eigenvalues ​​λ are all real numbers. If the eigenvalue corresponding to the minimum eigenvector is greater than 0, then the eigenvector is the estimated value of the surface normal vector, n = {n x ,n y ,n z}.

[0041] The beneficial effects of this method are:

[0042] This method uses a virtual fixture with motion constraints and force guidance to help the operator move the remote tool along the preferred path through enhanced guiding force, thereby improving remote operation performance, reducing the total workload and ensuring the safety of task execution; an autonomous agent is used to automatically maintain the angle and height of the welding, so that the operator only needs to perform 2 degrees of freedom control, thereby reducing the operator's mental burden and improving task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A system block diagram of an embodiment of the present invention.

[0044] Figure 2 It is a main flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0046] like Figure 1 As shown, the remote laser repair method based on virtual fixture and automatic agent described in the present invention,

[0047] The master system is developed based on Unity3D, connected to VR devices and hand controllers. Users can view the reconstructed three-dimensional workpieces through VR devices and use hand controllers to select the laser repair path. The slave system is developed based on ROS, connected to the robot arm, laser welding gun, infrared camera and depth camera. The depth camera is used to collect 3D point cloud data of the workpiece to be repaired to reconstruct it in the master system; the infrared camera is used to collect the molten pool image of the laser repair to determine the current status; the robot arm and the laser welding gun are connected to perform real laser repair according to the robot arm posture and laser control information transmitted by the master.

[0048] Figure 2 The specific method of the embodiment is shown, which includes the following steps:

[0049] Step 1: Determine the reference direction and error direction of movement based on the virtual welding gun position input by the user:

[0050] Assume that the reference laser repair path selected by the operator is described by the parameter expression:

[0051] p(s)≡(x(s),y(s),z(s)) T ,s∈[a,b]

[0052] Where s is a parameter and [a,b] is the range of s. The current Cartesian position of the virtual welding gun at the master end is m c for:

[0053] m c =(x c ,y c ,z c ) T

[0054] When the master end virtual welding gun is not exactly located on the reference path, the point on the reference path closest to the current position It can be expressed as:

[0055]

[0056] Thus, the reference direction δ of the current point is defined p (m c ) and error direction e(m c )for:

[0057]

[0058] Where d represents the instantaneous motion direction vector along the path, is the normalized tangent vector at this point.

[0059] Step 2: Determine the magnitude of the master end guiding force and motion constraints:

[0060] When the virtual welding gun deviates from the reference path, the virtual fixture gives the operator force feedback to correct the error; at the same time, it gives the operator a driving force along the direction of the reference path, and finally gives the operator a resultant force F c :

[0061]

[0062] where F max is the maximum value of the error feedback force, F p is the driving force along the path, k e , k p In order to make the real repair path close to the reference path, the virtual welding gun position m with constraints is defined. w As actual input location:

[0063] m w =m c -k s e(m c )

[0064] where k s is with k e A positively correlated regulatory factor. s = 1 is a hard constraint, in which case the laser repair position is completely constrained and the operator can only move forward or backward along the reference path. <k sWhen <1, it is a soft constraint. At this time, the laser repair position will be close to the reference path, but the operator can also deviate from the reference path under force feedback.

[0065] Step 3: Determine the actual repair position of the slave end and the normal vector of the point:

[0066] In order to ensure that the weld spot actually falls on the workpiece surface, we use the projection of the virtual welding gun position on the workpiece surface as the real weld spot m r This function is implemented using the Raycast function of unity3d, recorded as:

[0067] m r =Raycast(m w )

[0068] The normal vector n of the point is obtained by the following method:

[0069] Fitting m based on the least squares method r The three-dimensional plane where the neighboring points pass through P i Mark, then the distance from the point to the plane can be defined as d i =(P i -A)·n, A represents the centroid of the points in the neighborhood, which is defined as:

[0070]

[0071] The least squares method is used to obtain d i = 0, that is, the surface normal vector n, which is calculated using principal component analysis. First, the covariance matrix C is constructed:

[0072]

[0073] By solving the eigenvalue λ and eigenvector V of C, we can find the surface normal vector n:

[0074]

[0075] Among them, i P i The weight value of the point, in this paper, all neighborhood points contribute the same amount to the normal vector estimation, and set ζ i =1. Since C is symmetric and semi-positive definite, its eigenvalues ​​λ are all real numbers. If the eigenvalue corresponding to the minimum eigenvector is greater than 0, then the eigenvector is the estimated value of the surface normal vector, n = {n x ,n y ,n z}.

[0076] Step 4: Determine the repair angle and height based on the current molten pool state and laser power:

[0077] The image sequence of the melt pool is represented as {I1,I2,...,I T}, where each image I t The size of is H×W×C; the laser power sequence is represented by {P1,P2,...,P T}.

[0078] For each time step t, we use CNN to extract the image I t Extract features from:

[0079] f t =CNN(I t )

[0080] The feature vector f extracted by CNN t With laser power P t Combine to form a merged feature vector:

[0081] z t =[f t ,P t ]

[0082] The merged feature sequence {z1,z2,...,z T} Input LSTM network for time series processing:

[0083] h t =LSTM(z t ,h t-1 )

[0084] The hidden state of LSTM is converted into the angle and height of the welding gun through the fully connected layer:

[0085] y t =[θ t ,h t ]=W y h t +b y

[0086] where y t is the output vector, W y is the weight matrix, b y is the bias vector.

[0087] Step 5: Use the autonomous agent to automatically determine the position of the slave robot arm:

[0088] With {δ p (m c ),n×δ p (m c ),n} as the laser repair point m r The local coordinate system of the point is obtained by weldThe optimal inclination angle and distance θ, h for laser repair, and the corresponding change matrix is ​​T θ,h , then the position of the end of the robot arm is T robot :

[0089] T robot =T weld T θ,h T tool -1

[0090] Where T tool is the transformation matrix from the end of the robot arm to the end of the welding gun. According to the posture of the end of the robot arm, the corresponding joint angle J is solved by inverse kinematics. virtual , and sends these angles to the robot arm controller to achieve the desired repair movement.

[0091] This embodiment uses a virtual fixture with motion constraints and force guidance to help the operator move the remote tool along the preferred path through enhanced guiding force, thereby improving remote operation performance, reducing the total workload and ensuring the safety of task execution; at the same time, an autonomous agent is used to automatically maintain the angle and height of the welding, so that the operator only needs to perform 2 degrees of freedom control, thereby reducing the operator's mental burden and improving task execution.

[0092] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A remote laser repair method based on virtual fixture and autonomous agent, characterized in that: The following steps are involved: 1) Reference direction and error direction of welding gun movement: The reference laser repair path selected by the operator is described by the parameter expression: p(s)≡(x(s),y(s),z(s)) T ,s∈[a,b] Where s is a parameter, [a,b] is the range of s; the current Cartesian position of the virtual welding gun at the master end is m c for: m c =(x c ,y c ,z c ) T When the master end virtual welding gun is not exactly located on the reference path, the reference direction δ of the point is defined p (m c ) and error direction e(m c )for: Where d represents the instantaneous motion direction vector along the path, is the point on the reference path closest to the current position, is the normalized tangent vector at the point; 2) Main end force guidance and motion constraint: When the virtual welding gun deviates from the reference path, the virtual fixture gives the operator force feedback to correct the error; at the same time, it gives the operator a driving force along the direction of the reference path, and finally gives the operator a resultant force F c : Among them, F max is the maximum value of the error feedback force, F p is the driving force along the path, k e , k p is the regulating factor; In order to make the real repair path close to the reference path, the virtual welding gun position m with constraints is defined w As actual input location: m w =m c -k s e(m c ) where k s is with k e A positively correlated regulatory factor; when k s = 1 is a hard constraint, in which case the laser repair position is completely constrained and the operator can only move forward or backward along the reference path; <k s When <1, it is a soft constraint. At this time, the laser repair position will be close to the reference path, but the operator can also deviate from the reference path under force feedback; 3) The actual repair position and normal vector from the end: In order to ensure that the repair point actually falls on the workpiece surface, the projection of the virtual welding gun position on the workpiece surface is used as the real laser repair point m r , recorded as: m r =Raycast(m w ) At the same time, the normal vector n of the point is calculated based on the point cloud data around the real laser repair point; 4) Determine the welding gun inclination angle and height: According to the image of the molten pool and the laser power during laser repair, the CNN-LSTM network is used to predict the optimal angle θ and height h of the welding gun; 5) The autonomous agent determines the position of the slave robot arm: With {δ p (m c ),n×δ p (m c ),n} as the laser repair point m r The local coordinate system of the point is obtained by weld ; The optimal inclination angle and distance θ, h for laser repair, and the corresponding change matrix is ​​T θ,h , then the position of the end of the robot arm is T robot : T robot =T weld T θ,h T tool -1 Where T tool is the transformation matrix from the end of the robot arm to the end of the welding gun.

2. The remote laser repair method based on virtual fixture and autonomous agent according to claim 1 is characterized in that: It includes a master system and a slave system. The master system is developed based on Unity3D and is used to reconstruct the workpiece to be repaired and select the laser repair path. The slave system is developed based on ROS and connects the robotic arm, laser welding gun, infrared camera and depth camera, wherein the laser welding gun, infrared camera and depth camera maintain a rigid connection with the end of the robotic arm. The master system and the slave system realize data transmission through the 5G network.

3. The remote laser repair method based on virtual fixture and autonomous agent according to claim 1 is characterized in that: Step 3) The normal vector calculation method is: Fitting m based on the least squares method r The three-dimensional plane where the neighboring points pass through P i Mark, then the distance from the point to the plane can be defined as d i =(P i -A)·n, A represents the centroid of the points in the neighborhood, which is defined as: The least squares method is used to obtain d i = 0, that is, the surface normal vector n, which is calculated using principal component analysis. First, the covariance matrix C is constructed: By solving the eigenvalue λ and eigenvector V of C, we can find the surface normal vector n: Among them, i P i The weight value of the point, all neighboring points contribute the same amount to the normal vector estimation, set ζ i =1; since C is symmetric and semi-positive definite, its eigenvalues ​​λ are all real numbers. If the eigenvalue corresponding to the minimum eigenvector is greater than 0, then the eigenvector is the estimated value of the surface normal vector, n = {n x ,n y ,n z }.

4. The remote laser repair method based on virtual fixture and autonomous agent according to claim 1 is characterized in that: Step 4) determines the optimal angle and height of the welding gun according to the current molten pool state and laser power: The image sequence of the melt pool is represented as {I1,I2,...,I T }, where each image I t The size of is H×W×C; the laser power sequence is represented by {P1,P2,...,P T }; For each time step t, use CNN to extract the image I t Extract features from: f t =CNN(I t ) The feature vector f extracted by CNN t With laser power P t Combine to form a merged feature vector: z t =[f t ,P t ] The merged feature sequence {z1,z2,...,z T } Input LSTM network for time series processing: h t =LSTM(z t ,h t-1 ) The hidden state of LSTM is converted into the angle and height of the welding gun through the fully connected layer: y t =[θ t ,h t ]=W y h t +b y , where y t is the output vector, W y is the weight matrix, b y is the bias vector.

Images