System for enhancing teleoperation welding process telepresence

Abstract

The invention discloses a system for enhancing teleoperation welding process immediacy sense, which comprises a welding workbench for placing a weldment; the welding limb consists of a welding robot and a binocular vision camera fixed at the tail end of the welding robot; a welding gun is fixed at the tail end of the welding robot; the viewing limb consists of an auxiliary monitoring robot and two sets of molten pool vision cameras fixed at the tail end of the auxiliary monitoring robot; the tactile force feedback device is used for achieving action mapping between an operator and the welding robot and controlling the welding robot, the binocular vision camera, the molten pool vision camera and the VR glasses to be connected with the computer through teleoperation, and the computer converts pictures shot by the binocular vision camera into three-dimensional pictures to be displayed. Transmitting the data to the virtual reality glasses; and the virtual reality glasses are used for an operator to determine the arc starting position according to the displayed picture to carry out welding operation. The problem that teleoperation telepresence sense of welding operation is insufficient can be solved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and intelligent manufacturing, and in particular to a system and method for enhancing telepresence during a remote-controlled welding process. Background Art

[0002] Welding technology has been widely used in manufacturing industries such as automotive, shipbuilding, and aerospace. With the rapid development of the manufacturing industry in recent years, the requirements for welding technology have also gradually increased. The complexity and instability of the welding process can lead to product defects, equipment failures, and even serious accidents. Therefore, real-time monitoring and control of the welding process is critical to ensuring welding quality. Major national projects such as aerospace, nuclear energy, and space stations require welding operations in hazardous and extreme environments, such as space welding and welding in nuclear radiation sites. These environments are often beyond the reach of human operators, making the use of intelligent robots the most viable option. While current intelligent robotic welding systems have achieved intelligent control functions such as laser seam tracking and vision-guided trajectory planning, they still lack the ability to automatically adjust to welding process problems and lack flexibility in decision-making. Teleoperation is a more suitable control solution for complex working conditions. Lack of telepresence is a major technical challenge in current teleoperated welding technology. To address this issue, numerous researchers have conducted research on visualization of the teleoperation process. Synchronous display of the melt pool camera image and simulation are two important areas for remote operation visualization, but both have technical drawbacks. The melt pool camera display image has a single direction and is a two-dimensional image, which is significantly different from direct human visual observation. Simulation technology can only determine distance and position, but cannot display the specific situation of the melt pool process. MCAnanthram Rao et al. [Rao, MCA, Raj, S., Shah, AK et al. Development and comparison studies of XR interfaces for path definition in remote welding scenarios. Multimed Tools Appl 83, 55365–55404 (2024).] developed and compared XR interfaces (extended reality, a general term for virtual reality (VR), augmented reality (AR), and mixed reality (MR)) for remote welding scenarios. The study showed that compared with other XR technologies, VR interaction leads to more focused sensory perception and attention, and less behavioral pressure during interaction. However, the perspective of the VR scene constructed by the institute is relatively single. Although it has been proven that this technology is very suitable for path definition and curve tracking in the welding process, it cannot display the phenomena of the welding process in real time. It is only limited to the path planning needs in the horizontal plane.

[0003] In the existing papers and other documents, the lack of presence has always been a pain point. No scholar has used a completely convincing method to solve this problem. What is currently useful is to use VR to create a virtual reality scene. The constructed welding plate and welding gun are all digital models, but this virtual scene is usually not intuitive and has a large visual deviation from the actual situation. It cannot solve this pain point, but only gives people a reminder of the position and shape. Summary of the Invention

[0004] The purpose of the present invention is to provide a system for enhancing the telepresence of a remote-controlled welding process, so as to solve the problem of insufficient telepresence in remote-controlled welding operations.

[0005] The technical solutions for achieving the purpose of the present invention are:

[0006] A system for enhancing telepresence of a remotely operated welding process, comprising:

[0007] Welding workbench, used to place weldments;

[0008] A welding limb consisting of a welding robot and a binocular vision camera fixed at the end of the welding robot; a welding gun is fixed at the end of the welding robot; the binocular vision camera is used to capture real-time images of the welding gun and the weldment;

[0009] The viewing limb consists of an auxiliary monitoring robot and two sets of molten pool vision cameras fixed at the end of the auxiliary monitoring robot; the molten pool cameras are used to take real-time pictures of the molten pool during the welding process;

[0010] Tactile force feedback device is used to achieve motion mapping between the operator and the welding robot, so as to remotely control the welding robot.

[0011] The binocular vision camera, molten pool vision camera and virtual reality VR glasses are respectively connected to a computer. The computer converts the images captured by the binocular vision camera into a three-dimensional image display and transmits it to the virtual reality glasses; the virtual reality glasses are used by the operator to determine the arc starting position according to the displayed image to perform welding operations; the welding machine mode is switched to "arc starting", the binocular vision camera is turned off, the molten pool vision camera is turned on, and the image in the virtual reality glasses is switched to a real-time image of the molten pool vision; the operator operates the tactile force feedback device according to the molten pool vision image to control the welding robot to complete the welding operation.

[0012] Compared with the prior art, the present invention has the following significant advantages:

[0013] Currently, welding robots are still unable to automatically adjust to problems that arise during the welding process, and remote operation remains a more suitable control solution under complex working conditions. However, remote welding faces the technical difficulty of insufficient immersion. This invention uses a welding robot and a monitoring robot to coordinate with each other. Through the segmented coordination of binocular vision cameras and molten pool cameras, and the combination of dual molten pool camera image fusion and virtual reality technology, the welder can observe the changes in the welding pool image from multiple angles in a three-dimensional manner, increasing the welder's immersion and sense of presence during remote operation and improving the welder's accuracy in judging the process.

[0014] This invention proposes a system and method for enhancing telepresence during remote welding. The system includes coordinated collaboration between two robots, mapping the welder's teleoperation movements to the welding robot, mapping the welder's head posture to the auxiliary monitoring robot, designing a gaze and view drift mechanism for images in virtual reality glasses, and incorporating dual-pool camera stereo imaging technology. Ultimately, this enhances the telepresence of remote welding. This invention provides a new real-time, multi-angle monitoring method for intelligent industrial robot teleoperation, offering an immersive operating experience to remote operators, allowing them to more intuitively and accurately complete industrial tasks from a distance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of the system layout is provided.

[0016] Figure 2 A schematic diagram of the posture adjustment of the auxiliary monitoring robot end (melting pool camera).

[0017] Figure 3 This is the arrangement of two sets of melt pool cameras.

[0018] Figure 4 This is the magnification and regional drift of images in virtual reality (VR) glasses. DETAILED DESCRIPTION

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0020] A system for enhancing the sense of presence during a remote-controlled welding process of the present invention includes a welding robot 1, an auxiliary monitoring robot 2, a binocular vision camera 3, a binocular vision camera bracket 4, a molten pool vision camera 5, a molten pool vision camera bracket 6, a welding workbench 7, a computer, virtual reality (VR) glasses 8, a welding power source, a tactile force feedback device 9, a virtual reality monitoring system, a weldment 10, a welding gun 11, and other equipment.

[0021] Before performing remote welding operations, the equipment is assembled, wherein the binocular vision camera 3 is fixed to the end of the welding robot 1 through the binocular vision camera bracket 4, and the three form a welding limb 100. When installing the binocular vision camera, it is necessary to ensure that the camera can shoot in real time without obstructing the line of sight, so as to assist the welder in more accurate positioning of the weld before welding; two sets of molten pool vision cameras 5 are fixed to the auxiliary monitoring robot 2 through the molten pool vision camera bracket 6, and the three form a viewing limb 101. The molten pool camera 5 is used to shoot the molten pool image in real time during the welding process; the welding limb and the viewing limb are placed opposite each other on both sides of the welding workbench 7, and are both connected to the computer, and the weldment 10 is placed on the welding workbench 7. The binocular vision camera 3, the molten pool vision camera 5 and the virtual reality VR glasses 8 are respectively connected to the computer to transmit and display real-time images of the welding work area; the tactile force feedback device 9 is connected to the computer to realize the motion mapping between the operator and the welding robot 1, so as to remotely control the welding robot 1. The welding gun 11 is fixed to the end of the welding robot 1. The installation method of the entire set of equipment is as shown in the attached figure. Figure 1 As shown in the figure, the molten pool camera, visual camera, VR glasses, and supporting software and SDK of the robot are installed in sequence.

[0022] When preparing to start welding, the welding power supply is turned on, the welding machine mode is set to "arc extinguishing", the binocular vision camera 3 is turned on, and the camera captures a real-time image of the welding gun 11 and the weldment 10. The image is converted into a three-dimensional image display through the supporting software installed on the computer and transmitted to the virtual reality glasses 8; the welder wears the virtual reality (VR) glasses 8, determines the arc starting position according to the image displayed in the glasses, and performs the welding operation; the welding machine mode is switched to "arc starting", the binocular vision camera 3 is turned off, the molten pool vision camera 5 is turned on, and the image in the virtual reality glasses 8 is switched to a real-time image of the molten pool vision; the welder operates the tactile force feedback device 9 according to the molten pool vision image to control the welding robot 1 to complete the welding operation.

[0023] Based on the working principle of VR glasses, a dual-melt pool camera joint imaging system is built. Specifically, two sets of melt pool vision cameras 5 are placed in parallel to ensure that the optical axes of the cameras are parallel to simulate the parallax of the human eye. When installing the cameras, it is ensured that the melt pool vision cameras 5 can capture the complete melt pool image. The melt pool vision cameras 5 are set to maintain the same frame rate, and the on and off trigger events of the melt pool vision cameras 5 are set to ensure that the two melt pool images are read synchronously (time error <1ms). Based on Zhang Zhengyou calibration and Fusiello correction, a binocular stereo vision system is constructed. After image preprocessing, SGM stereo matching and 3D reconstruction, high-precision digital modeling of the melt pool morphology is achieved. Finally, a side-by-side split-screen video is generated and transmitted to the VR glasses.

[0024] The control strategy for the two robots is to separate and prioritize degrees of freedom for the auxiliary monitoring robot 2, with the robot's end-point posture set as the first priority and displacement as the second. The displacement degrees of freedom are computer-controlled. The auxiliary monitoring robot 2, equipped with a weld pool vision camera 5, follows the welding robot 1 in a direction parallel to the weld seam. During the welding process, the weld pool vision camera 5 ensures that the welding torch 11 and the weldment 10 are captured within its field of view, allowing for clear weld pool images. The rotational degrees of freedom are controlled by the operator. The auxiliary monitoring robot 2 performs angle mapping based on the operator's head posture, adjusting the end-point posture in real time to ensure that the weld pool vision camera 5's shooting angle displayed in the VR glasses reflects the welder's intended viewing area. All degrees of freedom of the welding robot 1 are controlled by the operator using a tactile force feedback device 9. The welding operation of the welding robot 1 is achieved by designing a mapping relationship between the force feedback device and the coordinates of the welding robot 1.

[0025] The image in the virtual reality glasses 8 is processed, and a gaze and view drift mechanism is introduced to mark the position where the human eye is looking at the image, and the coordinate difference between the gaze point and the center point of the displayed image is calculated. The local area is magnified by gaze, which improves the accuracy of the operator's judgment of the phenomenon. The view drift mechanism ensures that the gaze position is always located at the center of the image displayed in the virtual reality glasses 8, that is, the gaze point coincides with the center of the image.

[0026] An SDK for welding equipment was developed, and the robot's control mode was divided into two states: "arc on" and "arc off," depending on the welding process. "Arc off" is the default state. The welding power supply remains on. When in the "arc on" state, the power supply provides current and simultaneously starts feeding gas and wire. The ZED binocular vision camera 3 remains off, while the weld pool vision camera 5 remains on. When in the "arc off" state, the current, protector, and filler wire are stopped, the ZED binocular vision camera 3 is turned on, and the weld pool vision camera 5 is turned off. The two states are switched using a button on the Touch 6DOF handheld robotic arm.

[0027] In this example, the tactile force feedback device 9 uses a Touch 6-DOF robot, and the welding robot 1 and the auxiliary monitoring robot 2 use KUKA robots. The mapping relationship between the tactile force feedback device 9 and the welding robot 1 is position mapping, which synchronizes the position coordinates of the end of the Touch 6-DOF robot's joystick with the end of the welding robot's welding gun. The spatial coordinates of the Touch 6-DOF robot's joystick are P touch (x touch ,y touch , z touch ), Touch the six-degree-of-freedom robot coordinate system (the robot has its own world coordinate system) origin coordinate O touch (0 touch , 0 touch , 0touch ); the spatial coordinate of the end of the welding robot's welding gun is P weld (x weld ,y weld , z weld ), the origin coordinate of the welding robot coordinate system (the robot's own world coordinate system) is O weld (0 weld , 0 weld , 0 weld ), after position synchronization, when the end of the rocker of the Touch 6-DOF robot is at the coordinate O of the origin of the Touch 6-DOF robot coordinate system touch (0 touch , 0 touch , 0 touch ), the end of the welding robot's welding gun is also at the origin of its coordinate system (0 weld , 0 weld , 0 weld In this example, the coordinates of the Touch 6DOF robot are aligned with those of the welding robot 1. The auxiliary monitoring robot 2 is placed face to face with the welding robot 1 and has a mirror image relationship with the Touch 6DOF robot in the x-direction.

[0028] The data between the tactile force feedback device 9 and the welding robot 1 is transmitted through ROS2 communication. In order to reduce the impact of unintentional behaviors such as muscle tremors when the operator uses the Touch six-degree-of-freedom robot on welding accuracy, a data processing module is added during the data transmission process to perform the following processing: Taking into account the physiological characteristics of human muscle tremors, the tremor frequency is usually 8 to 12 Hz, while the frequency of normal voluntary behavior is less than 6 Hz. The time domain signal of the human hand received by the Touch six-degree-of-freedom robot is converted into a frequency domain signal through fast Fourier transform (FFT) and low-pass filtering (LPF) is performed. The frequency threshold is set to 6 Hz, and the signal is filtered using the filtfilt function in the scipy library of python. This function can avoid the disadvantage of low-pass filtering phase delay by passing the low-pass filter twice in the forward and reverse directions. The filtered signal may have a small number of singular points. Considering the continuity of the welding action, the exponentially weighted moving average (EWMA) algorithm is added in this example. Its algorithm formula is shown in formula (1). Through this algorithm, the instant action is connected with the action of the previous timestamp, which increases the smoothness of the action. The spatial coordinates of the end of the rocker of the Touch 6DOF robot processed by the above algorithm in the Touch 6DOF robot coordinate system are P filterd (x filterd ,yfilterd , z filterd ), P filterd The coordinates are passed to the welding robot through ROS2 to execute motion instructions in real time.

[0029] EWMA t =αP t +(1-α)EWMA t-1 (1)

[0030] EWMA t is the optimized coordinate at the current time t, α is the smoothing coefficient, which is used to control the balance between the current time data and the EWMA value of the previous time t-1; P t is the measured coordinate at the current time t.

[0031] The mapping relationship between the two is divided into two cases according to different intentions. When switching to the "arc start" mode, the mapping relationship is 1:2, and the displacement of the welding robot 1 is twice the displacement of the hand, that is, The left side of the equation represents the distance from the end of the welding robot's welding torch to the robot's spatial origin. The right side represents twice the distance from the end of the Touch 6DOF robot's joystick to the origin of the Touch 6DOF robot's coordinate system. In this mode, the welding robot's welding torch moves slowly, facilitating fine adjustments. When switched to "arc extinguishing" mode, the mapping is 1:10, resulting in faster robot travel and facilitating rapid positioning. The robot's yaw, pitch, and roll angles remain consistent with those of the Touch robot's end (and the operator's hand movements). The robot employs a speed anomaly protection mechanism that disables activation when the speed exceeds 30 cm / s to prevent damage to the robot caused by improper operation.

[0032] During welding, the operator wears virtual reality glasses 8 to observe the video image transmitted by the camera. In this example, a PICO VR all-in-one device is used, which provides accurate eye tracking. When the welder is in "arc off" mode, the binocular vision camera 5 remains on. In this example, the selected device is a ZED binocular vision camera, which captures a real-time image of the welding torch 11 and the weldment 10. The image is converted into a three-dimensional depth image using supporting software installed on the computer and transmitted to the VR glasses. The operator observes the image and controls the Touch 6DOF robot end to guide the welding torch 11 to the target arc starting position. When the welder is switched to "arc starting" mode, the binocular vision camera 3 is turned off, the molten pool vision camera 5 is turned on, and the molten pool vision camera image is transmitted to the virtual reality glasses 8.

[0033] The auxiliary monitoring robot 2 adopts a degree of freedom separation and priority control method. The first control priority is the auxiliary monitoring robot terminal posture, that is, the yaw angle, pitch angle, and roll angle degrees of freedom. The second control priority is the x, y, and z direction movement degrees of freedom. On the basis of ensuring the accuracy of the posture, the moving path is automatically calculated. Before starting welding, the postures and relative positions of the two robots are initialized according to the parameters of the molten pool vision camera 5 to ensure that a clear real-time picture of the molten pool can be captured, and the distance from the center of the lens of the molten pool vision camera 5 to the welding gun is controlled to a fixed value, which is set to 25cm in this example. The posture change of the terminal of the auxiliary detection robot 2 is controlled by the virtual reality glasses 8 (operator's head posture) and is consistent with the coordinate system of the virtual reality glasses 8. When the operator wants to observe from the left side of the molten pool, he turns his head, and the angle of the virtual reality glasses 8 changes from (0,0,0) to (VR Yaw , VR Pitch , VR Roll ), VR Yaw , VR Pitch , VR Roll They are the yaw angle (Yaw), pitch angle (Pitch) and roll angle of the virtual reality glasses 9. The angle changes are mapped to the auxiliary monitoring machine 2, causing the deflection of the molten pool visual camera 5. When the posture of the molten pool visual camera 5 is synchronized to (VR Yaw , VR Pitch , VR Roll ), the computer automatically calculates the position where the molten pool vision camera 5 should be located, that is, in the coordinate system of the auxiliary monitoring robot 2, with the welding gun end 11 as the starting point and the direction (VR Yaw , VR Pitch , VR Roll ) is on the straight line, marked as point A. The auxiliary monitoring robot 2 system performs inverse kinematics solution to guide the auxiliary monitoring robot 2 end to move to the position of target point A while maintaining the same posture. At this time, a clear image of the molten pool is generated in the glasses under the left-biased angle. The adjustment process is shown in the attached figure. Figure 2 shown.

[0034] The screen of the virtual reality glasses 8 uses polarization technology to separate images, providing independent images for the left and right eyes, and relying on the human brain's physiological fusion to produce a sense of three-dimensionality. Currently, all melt pool images are displayed flat, lacking three-dimensionality and spatial perception. Based on the working principle of the virtual reality glasses 8, this invention designs a dual-melt pool camera combined imaging system. The melt pool images captured by the two cameras are sequentially subjected to line of sight correction, depth map conversion, and VR-adaptive rendering before being transmitted to the virtual reality glasses 8 to generate a three-dimensional melt pool image.

[0035] Hardware preparation: Two sets of molten pool vision cameras 5 are installed side by side, with the same height to prevent longitudinal parallax; the optical axes of the molten pool vision cameras 5 are parallel, and the baseline distance T is set. xThe default initial angle between the camera optical axis and the molten pool plane is 30° to reduce the interference of mirror reflection. The molten pool vision camera 5 is designed to start synchronously after the time signal triggers the molten pool camera. The camera acquisition frequency is kept consistent so that the exposure event deviation of the two cameras is less than 0.1ms. The camera is equipped with a narrow-band filter to reduce arc interference. The distance from the camera to the molten pool along the optical axis is 25cm to ensure that both cameras can fully capture the complete molten pool image. Figure 3 shown.

[0036] Calculation process:

[0037] First, use the Zhang Zhengyou calibration method to shoot the checkerboard calibration plate with two left and right molten pool vision cameras 5 to obtain 40 sets of clear images. These images need to cover various working angles of the camera to ensure the accuracy of the calibration. Use the corner detection function in the Opencv library to process the collected checkerboard image and detect the inner corner points of the checkerboard. The correspondence between the image coordinates and the world coordinates is established through the pixel coordinates of these corner points in the image and their corresponding world coordinates. According to the principle of Zhang Zhengyou calibration method, a set of equations containing the camera intrinsic parameter matrix K, the extrinsic parameter matrix (rotation matrix R and translation vector T) and the distortion coefficient is constructed. The camera intrinsic parameter matrix K is in the form of:

[0038]

[0039] Among them, f x and f y is the focal length of the camera in the x and y directions, c x and c y is the principal point coordinate of the image, and s is the axis tilt parameter (the degree of tilt between the x and y axes). The extrinsic matrix is ​​used to describe the rotation and translation relationship between the camera coordinate system and the world coordinate system.

[0040] Distortion coefficients include radial distortion coefficients [k1, k2, k3] and tangential distortion coefficients [p1, p2]. Radial distortion is caused by imperfections in the lens shape, resulting in barrel or pincushion-shaped image deformation; tangential distortion is caused by the lens being mounted nonparallel to the imaging plane. The above equations (including the camera intrinsic parameter matrix K, extrinsic parameter matrices R and T, and the distortion coefficient equations) are solved using optimization algorithms such as least squares to obtain the intrinsic and extrinsic parameters of the left and right melt pool cameras and the distortion coefficients of the original image.

[0041] Based on the obtained distortion coefficient, the original image is corrected using the distortion correction formula. For each pixel (x, y) in the image, radial correction and tangential correction are performed in sequence according to the following formula:

[0042] Radial distortion correction:

[0043]

[0044] x radial 、y radial are the horizontal and vertical coordinates of the pixel after radial correction, and r is the normalized polar diameter of the pixel in the image plane, that is, the distance from the pixel to the center point (principal point) of the image.

[0045] Tangential distortion correction:

[0046]

[0047] x distorted 、y distorted are the horizontal and vertical coordinates of the pixel after tangential correction.

[0048] in, After correction, a distortion-free image is obtained, providing an accurate data basis for subsequent processing.

[0049] Use the Fusiello correction method to calibrate the epipolar lines: Calculate the basic matrix F: In the two corrected images, find a series of matching point pairs (p l ,p r ), where p l is the midpoint of the left image, p r are the corresponding points in the right figure. According to the principle of epipolar geometry, these matching point pairs satisfy the basic matrix constraints:

[0050]

[0051] Using methods such as the eight-point algorithm, the fundamental matrix F is solved through at least 8 sets of matching point pairs. The fundamental matrix F describes the epipolar geometric relationship between the left and right images, which includes the rotation, translation, and intrinsic parameter information between the images.

[0052] Calculate the essential matrix E: the intrinsic parameter matrix K of the left and right cameras is known l and K r , through the formula Calculate the essential matrix E. The essential matrix E can be decomposed into a combination of the rotation matrix R and the translation vector t (essential matrix parameter, representing the translation vector between the two cameras), that is, E = [t] × R, where [t] × is the antisymmetric matrix of the translation vector t.

[0053] Solve the rotation matrix R and translation vector t: Perform singular value decomposition (SVD) on the essential matrix E: E = U∑V T, where U and V are orthogonal matrices, and ∑ is a diagonal matrix. By processing the singular value decomposition results, four possible (R, t) combinations are obtained. Based on the actual camera position and scene information, the set of (R, t) that matches the actual situation is selected as the final rotation matrix and translation vector.

[0054] Calculate the correction transformation matrix H for the left and right cameras l and H r :According to the obtained rotation matrix R and translation vector t, as well as the intrinsic parameter matrix K of the left and right cameras l and K r , calculate the correction transformation matrix H of the left and right cameras l and H r :

[0055]

[0056] Among them, R l is the rotation matrix of the left camera, R r is the rotation matrix of the right camera.

[0057] Use the correction transformation matrix H of the left and right cameras l and H r The left and right images are transformed separately so that the rectified images satisfy the horizontal epipolar constraint, that is, the ordinates of corresponding points in the left and right images are equal, thereby simplifying the subsequent stereo matching process.

[0058] Preprocessing of stereo images: LOG operator image noise filtering: Laplacian of Gaussian (LOG) operator is a commonly used image filtering method that can effectively detect edges and noise in images. The expression of the LOG operator is:

[0059]

[0060] in, The Laplacian operator is a second-order derivative operator used to detect rapid changes in grayscale values ​​in an image. G(x, y, σ) is the Gaussian kernel function, and σ is the scale parameter. By adjusting the value of σ, you can control the intensity of the filter and its response to features of different scales. The LOG Laplacian of Gaussian operator is convolved with the image to produce a filtered image, removing noise while preserving edge information.

[0061] Histogram equalization eliminates brightness differences: Due to differences in left and right camera parameters and the influence of the shooting environment, there may be brightness differences between the left and right images. Histogram equalization is a commonly used image enhancement technology. It redistributes the grayscale values ​​of the image pixels to make the grayscale distribution of the image more uniform, thereby improving the contrast and visual effect of the image. Its formula is:

[0062]

[0063] Here, h(j) is the original histogram, N is the total number of pixels, and h'(i) is the new grayscale image, which is the grayscale image after increasing the contrast. The round() function rounds the image to an integer, and i is a threshold. The calculation here represents the percentage of pixels in the image with a grayscale value less than or equal to i.

[0064] Stereo matching using the SGM algorithm: Cost calculation: The semi-global matching (SGM) algorithm first performs cost calculation. For each pixel p and different disparity values ​​d at the left image terminal, the matching cost C(p,d) is calculated. The matching cost usually consists of two parts:

[0065] C(p,d)=SSD(p,d)+λ·CT(p,d)

[0066] Among them, SSD calculates the square difference of the grayscale of pixel blocks, CT performs domain transformation to resist light interference, and λ balances the weights of the two.

[0067] One-dimensional energy optimization: in multiple directions Press up Aggregation cost, P1 and P2 control the parallax smoothness.

[0068] in is the cumulative cost value when the disparity is d at pixel p; d ' Indicates the previous pixel Possible disparity value in the incoming aggregation path, min d' It means that after traversing all possible disparities, the disparity with the minimum energy is selected as the optimal disparity; d" represents the disparity value with the minimum aggregation cost among all disparities at the previous pixel point; min d” () is to find the optimal disparity in the neighborhood; is a normalization term to avoid cost accumulation overflow; T(|d-d'|=1) is a penalty switch. When the condition dd is satisfied ' When |=1, the corresponding penalty is activated; the P1 and P2 coefficients are used to control the parallax smoothness;

[0069] Disparity map generation: Accumulate the aggregation cost in each direction The d that minimizes S(p,d) is taken as the disparity value of pixel p to form a disparity map.

[0070] Disparity map median filtering: Median filtering is a nonlinear filtering method that removes noise and outliers by replacing the grayscale value of each pixel in the image with the median of the grayscale values ​​of the pixels in its neighborhood. For each pixel p in the disparity map, a neighborhood N(p) of size n×n (such as 3×3 or 5×5) is selected with p as the center, and the median of the disparity values ​​of all pixels in the neighborhood is calculated, and this median is used as the new disparity value of pixel p:

[0071] d'(p)=median{d(q)|q∈N(p)}

[0072] Where median is the median function, d(q) is the pixel disparity, and this formula is the median of the disparity values ​​of all pixels in the neighborhood.

[0073] Through median filtering, isolated noise points and disparity points inconsistent with the surrounding areas in the disparity map can be effectively removed, making the disparity map smoother and more accurate, and improving the quality of subsequent 3D reconstruction.

[0074] Three-dimensional reconstruction using local fitting method: According to the formula Where Z is the scene depth, which is calculated by the parallax value of the melt pool vision camera, the focal length f and the baseline distance B. The pixel points in the image collected by the camera are transformed into coordinates by Convert pixel coordinates to camera normalized coordinates to form preliminary 3D point cloud coordinates, where u and v are the actual pixel values ​​in the horizontal and vertical directions, and x and y are the actual values ​​scaled to numbers between 0 and 1, which is a proportional relationship. Then perform local fitting on the preliminary point cloud. The specific method is: for each point p, find k nearest neighbor points p i ,use Fitting, where i represents the serial number, ranging from 1 to k, w(p,p i ) represents the i-th adjacent point p i The contribution weight of the fitting of the center point p, f(p) is the fitting function value of the target point p; k is the number of neighborhood points, (a0+a1x+a2y+a3z) represents the local surface where point p is located, x, y, z are the coordinates of point p in the camera coordinate system. w is the Gaussian weight function, and the mathematical expression is: ||pp i || is the adjacent point p i The Euclidean distance from p, a0, a1, a2, a3 are coefficients, solved by the least squares method, by locally fitting all points, the discrete three-dimensional data information is connected into a continuous three-dimensional entity, and then Convert to the world coordinate system, complete the 3D reconstruction, and form a regular 3D point cloud image. w Y w Z w] is the coordinate of this set of scene points in the world coordinate system, [X c Y c Z c ] is the coordinate of a point in the scene in the molten pool visual camera coordinate system, which is calculated by combining the normalized coordinates with the depth Z. c =x·Z,Y c =y·Z,Z c =Z;

[0075] The reconstructed 3D image is transmitted to the host computer, where it is analyzed and processed to identify and track the weld pool. Simultaneously, the 3D image is transmitted to a VR device, providing the operator with a immersive, depth-informed view, enabling a more intuitive observation of the weld pool and precise control of welding operations.

[0076] Virtual reality glasses 8 gaze mechanism: Introducing the attention mechanism into the image of virtual reality glasses 8, in this example, virtual reality glasses 8 can obtain eye movement data in real time, that is, the coordinates of the line of sight projected onto the screen are marked as point gaze (x gaze ,y gaze ), this data records the position of the eye gaze image. When the gaze point is concentrated on a small area of ​​the melt pool image (in this example, the small area is set to a square area with a ratio of 0.1×0.1 to the size of the global melt pool image) for more than 2 seconds, it is marked as gaze, and the local image of the melt pool is gradually enlarged with the last moment of gaze as the center until it covers the entire visual area of ​​the virtual reality glasses 8. The local image content is an area with a ratio of 0.4×0.4 to the global melt pool image, so that the operator can more accurately detect the changes in the melt pool in the area of ​​interest. As shown in the attached figure Figure 4 In the figure, the initial image is the entire melt pool image. When the gaze is focused on the area around circle ① for more than 2 seconds, the local area is enlarged, and the sight area ① (the solid line frame in the figure) fills the field of view of the VR glasses, and the local real-time image replaces the overall image.

[0077] Virtual reality glasses 8 View drift mechanism: In the gaze state, if the distance between the gaze point and the center of the current local area is less than 0.2 of the local image size ratio, the view drift mechanism is not activated and the view area remains unchanged; when the distance between the gaze point and the center of the current local area accounts for 0.2-0.8 of the local image size ratio (view drift condition), the view drift mechanism is activated and the local area changes, such as the attached Figure 4In the figure, the operator looks at the circle ② in the sight area ①. The distance meets the view drift condition, and the local view moves to the sight area ② (the local view area with ② as the center and the same size as the sight area ①), which is the dotted box in the figure. If the distance between the gaze point and the center of the current local area exceeds 0.8 of the local image size ratio, the gaze point quickly moves to the boundary of the local view, and the intention is recognized as changing the focus area. The gaze mechanism is released, and the melt pool image in the VR glasses gradually recovers to the global melt pool image.

[0078] The welder remotely controls the welding operation: Turn on the welding power supply, and the welder is in the "arc extinguishing" state. The welder wears VR glasses and remotely controls the welding robot to the optimal arc starting position based on the binocular vision camera image. Press the button on the Touch remote sensor, and the welder switches to the "arc starting" state to start the welding operation. At this time, the image in the VR glasses is a real-time image of the molten pool. The welder completes the welding operation by changing posture, gazing, and other methods, and using the designed line of sight gaze and view drift mechanism to determine the optimal observation angle and key observation area.

Claims

1. A system for enhancing telepresence during remote welding, characterized in that: include: Welding workbench, used to place weldments; A welding limb consisting of a welding robot and a binocular vision camera fixed at the end of the welding robot; a welding gun is fixed at the end of the welding robot; the binocular vision camera is used to capture real-time images of the welding gun and the weldment; The viewing limb consists of an auxiliary monitoring robot and two sets of molten pool vision cameras fixed at the end of the auxiliary monitoring robot; the molten pool cameras are used to take real-time pictures of the molten pool during the welding process; Tactile force feedback device is used to achieve motion mapping between the operator and the welding robot, so as to remotely control the welding robot. The binocular vision camera, molten pool vision camera and virtual reality VR glasses are respectively connected to a computer. The computer converts the images captured by the binocular vision camera into a three-dimensional image display and transmits it to the virtual reality glasses; the virtual reality glasses are used by the operator to determine the arc starting position according to the displayed image to perform welding operations; the welding machine mode is switched to "arc starting", the binocular vision camera is turned off, the molten pool vision camera is turned on, and the image in the virtual reality glasses is switched to a real-time image of the molten pool vision; the operator operates the tactile force feedback device according to the molten pool vision image to control the welding robot to complete the welding operation.

2. The telepresence enhancement system for remote welding process according to claim 1, characterized in that: The auxiliary monitoring robot adopts degree of freedom separation and priority differentiation control: the terminal posture of the auxiliary monitoring robot is set as the first priority, and the displacement is the second priority; the displacement degree of freedom is controlled by a computer, and the auxiliary monitoring robot with a fixed molten pool vision camera follows the welding robot and moves in a direction parallel to the weld. During the shooting of the welding process, the molten pool vision camera shooting area includes the welding gun and the weldment to obtain the molten pool image; the rotational degree of freedom is controlled by the operator, and the auxiliary monitoring robot performs angle mapping according to the operator's head posture, and adjusts the terminal posture in real time to ensure that the shooting angle of the molten pool vision camera displayed in the virtual reality glasses is the welder's intended observation area; all degrees of freedom of the welding robot are controlled by the operator through a tactile force feedback device, and the welding operation of the welding robot is realized by designing a mapping relationship between the force feedback device and the welding robot coordinates.

3. The telepresence enhancement system for remote welding process according to claim 1, characterized in that: The welding limb and the monitoring limb are placed opposite to each other on both sides of the welding workbench; the mapping relationship between the tactile force feedback device and the welding robot is position mapping, the coordinates of the tactile force feedback device are consistent with the coordinate direction of the welding robot, and the auxiliary monitoring robot and the welding robot are placed face to face, and there is a mirror relationship in the x-direction with the coordinates of the tactile force feedback device.

4. The system for enhancing telepresence of remote welding process according to claim 1, characterized in that: The computer is provided with a data processing module: the time domain signal of the human hand received by the tactile force feedback device is converted into a frequency domain signal by fast Fourier transform and low-pass filtered, the filtered signal is connected with the action of the previous timestamp by exponential weighted moving average algorithm, and the spatial coordinate P of the end of the joystick of the tactile force feedback device is converted into a frequency domain signal by fast Fourier transform and low-pass filtered. filterd The motion instructions are transmitted to the welding robot for real-time execution. When switched to the "arc start" mode, the mapping relationship is 1:2, and the welding robot's displacement is twice the hand's displacement, that is, When switched to the "arc extinguishing" mode, the mapping relationship is 1:10; weld is the spatial coordinate of the end of the welding robot’s welding gun, O weld is the origin coordinate of the welding robot coordinate system, O touch is the origin coordinate of the tactile force feedback device coordinate system.

5. The system for enhancing telepresence of remote welding process according to claim 1, characterized in that: Two sets of melt pool vision cameras are placed in parallel, at the same height, with parallel optical axes and consistent acquisition frequency, so that they can both capture the complete melt pool image. The computer forms a three-dimensional image based on the two sets of melt pool vision cameras: First, the Zhang Zhengyou calibration method is used to obtain the intrinsic parameter matrix K, extrinsic parameter matrix and distortion coefficient of the melt pool vision camera; According to the obtained distortion coefficient, the original image is corrected using the distortion correction formula; The epipolar lines were calibrated using the Fusiello correction method; Calculate the essential matrix, perform singular value decomposition on the essential matrix E, and solve the rotation matrix and translation vector; Calculate the correction transformation matrix of the left and right cameras based on the obtained rotation matrix and translation vector; use the correction transformation matrix of the left and right cameras and transform the left and right images respectively so that the corrected images meet the horizontal epipolar constraint; Preprocess the image to obtain the filtered image Histogram equalization is used to eliminate the brightness difference between the left and right images; The SGM algorithm is used for stereo matching, and the local fitting method is used for 3D reconstruction.

6. The telepresence enhancement system for remote welding process according to claim 5, characterized in that: Use the SGM algorithm for stereo matching and the local fitting method for 3D reconstruction, including: First, the cost is calculated. For each pixel p and different disparity values ​​d at the left image terminal, the matching cost C(p,d) is calculated: C(p,d)=SSD(p,d)+λ·CT(p,d) Among them, SSD means calculating the square difference of grayscale of pixel blocks, CT means performing domain transformation to resist light interference, and λ means weight; One-dimensional energy optimization: in multiple directions Press up Aggregation cost; in It is the cumulative cost value when the disparity is d when aggregated at pixel p; d' represents the total cost from the previous pixel Possible disparity value in the incoming aggregation path, min d' It means that after traversing all possible disparities, the disparity with the minimum energy is selected as the optimal disparity; d" represents the disparity value with the minimum aggregation cost among all disparities at the previous pixel point; min d” () is to find the optimal disparity in the neighborhood; It is a normalization term to avoid overflow of cost accumulation; T(|d-d'|=1) is a penalty switch. When the condition |d-d'|=1 is met, the corresponding penalty is activated; P1 and P2 coefficients are used to control the parallax smoothness; Disparity map generation: Accumulate the aggregation cost in each direction Take the d that minimizes S(p,d) as the disparity value of pixel p to form a disparity map; Disparity map median filtering: For each pixel p in the disparity map, select a neighborhood N(p) of size n×n with p as the center, calculate the median of the disparity values ​​of all pixels in the neighborhood, and use the median as the new disparity value d'(p) of pixel p; 3D reconstruction using local fitting method: According to Calculate the depth Z of the scene point, where f and B are the focal length and baseline distance of the molten pool vision camera; perform coordinate transformation on the pixel points in the image captured by the camera, convert the pixel coordinates to the normalized camera coordinates to form the initial three-dimensional point cloud coordinates, and then perform local fitting on the initial point cloud. The specific method is: for each point p, find k nearest neighbor points p i ,use Fitting, w(p,p i ) represents the i-th adjacent point p i The contribution weight of the fitting of the center point p, f(p) is the fitting function value of the target point p; k is the number of neighborhood points, (a0+a1x+a2y+a3z) represents the local surface where point p is located, x, y, z are the coordinates of point p in the molten pool visual camera coordinate system, w is the Gaussian weight function, a0, a1, a2, a3 are coefficients; by performing local fitting on all points, the discrete 3D data information is connected into a continuous 3D entity, and then Convert to the world coordinate system, complete the 3D reconstruction, and form a regular 3D point cloud image; where [X w Y w Z w ] is the coordinate of this set of scene points in the world coordinate system, [X c Y c Z c ] is the coordinate of a point in the scene in the molten pool visual camera coordinate system, which is calculated by combining the normalized coordinates with the depth Z. c =x·Z,Y c =y·Z,Z c =Z.

7. The telepresence enhancement system for remote welding process according to claim 1, characterized in that: The virtual reality glasses adopt the gaze mechanism and view drift mechanism: the virtual reality glasses can obtain eye movement data in real time, that is, the coordinates of the line of sight projected onto the screen are marked as point gaze (x gaze ,y gaze When the gaze point is focused on a certain micro-area of ​​the melt pool image for longer than the set time, it is marked as gaze, and the local image of the melt pool is gradually enlarged with the last gaze point as the center until it covers the entire visible area of ​​the virtual reality glasses; In the gaze state, if the distance between the gaze point and the center of the current local area is less than the set local image size ratio, the view drift mechanism is not activated and the view area remains unchanged; when the distance between the gaze point and the center of the current local area accounts for the set local image size ratio, the view drift mechanism is activated, the gaze point moves to the local view boundary, the recognition intention is to change the focus area, the gaze mechanism is released, and the melt pool image in the virtual reality glasses gradually returns to the global melt pool image.

8. The telepresence enhancement system for remote welding process according to claim 1, characterized in that: The tactile force feedback device uses a Touch six-degree-of-freedom robot.

Images