Welding robot work monitoring method and system based on machine vision

Abstract

The invention discloses a welding robot work monitoring method and system based on machine vision. According to the scheme, a monitoring platform composed of an industrial robot, a welding head, a machine vision sensor, an industrial personal computer and a controller is constructed; on the basis, a unified coordinate system is established through camera internal reference calibration, hand-eye calibration and tool center point calibration; welding images, robot poses and process parameters are collected, and welding seam features are extracted through preprocessing and dynamic region of interest locking; performing three-dimensional reconstruction and track fitting by combining the depth information to generate a welding seam reference track and a welding gun theoretical attitude; and finally, the working state is judged by fusing the deviation parameters, the track and the technological parameters, time delay compensation, online deviation correction and abnormal alarm are carried out, and the scheme can improve the welding seam monitoring precision, the tracking stability and the welding quality consistency under the strong interference working condition.

Description

Technical Field

[0001] This invention relates to intelligent manufacturing technology and welding technology, and in particular to a method and system for monitoring the operation of welding robots based on machine vision. Background Technology

[0002] Welding robots, as an important component of intelligent manufacturing equipment, have been widely applied in automotive manufacturing, construction machinery, rail transportation, new energy battery components, and the connection of high-end equipment parts. In particular, laser welding and high-precision arc welding processes offer advantages such as low heat input, high welding speed, high forming quality, and ease of automation integration, thus placing higher demands on welding robots in terms of trajectory accuracy, attitude control, and process stability. The appendix also clearly points out that, against the backdrop of the digital and intelligent upgrading of manufacturing, vision-guided welding has become an important direction for improving welding quality and efficiency.

[0003] Most existing welding robots employ a teach-and-playback method for welding operations, where a human first teaches the robot to obtain the welding trajectory, and then the robot repeatedly executes the preset path. This method is feasible for regular workpieces and stable operating conditions, but it struggles to adapt promptly to common issues in actual production, such as workpiece assembly errors, weld misalignment, gap fluctuations, workpiece thermal deformation, and tooling repetitive positioning errors. Especially in high-precision welding scenarios, where the welding spot size is small and the alignment accuracy between the welding torch and the weld is extremely high, relying solely on offline or taught trajectories can easily lead to problems such as weld misalignment, weld gaps, and inconsistent weld formation. These limitations are clearly discussed in the appendix.

[0004] To address the aforementioned issues, existing technologies have begun to incorporate visual sensors for weld seam identification and tracking. However, current visual monitoring solutions still suffer from the following shortcomings: First, passive vision methods are susceptible to the effects of arc light, smoke, spatter, and high reflectivity of the workpiece, making it difficult to stably acquire high signal-to-noise ratio images. Second, while some image processing methods can extract two-dimensional weld seam features, they struggle to accurately reflect the spatial location of the weld seam and the relative posture of the welding torch. Third, although some deep learning-based recognition methods improve anti-interference capabilities, they are heavily reliant on the scale of training data and computational resources, limiting their real-time performance in industrial settings. Fourth, existing technologies often focus more on weld seam identification or trajectory generation itself, lacking a unified monitoring and grading mechanism for lateral deviations, height deviations, posture deviations, visual distortions, and process anomalies during welding robot operations. The attached text's analysis of the current status and bottlenecks of weld seam visual recognition, 3D point cloud processing, and trajectory planning precisely supports the identification of this technical problem.

[0005] On the other hand, even if existing technologies can obtain weld seam images or local point clouds, practical applications still generally suffer from the superposition of visual acquisition latency, image processing latency, communication latency, and robot inertial response latency. In other words, the deviations detected by the system are often past events, while the robot's action adjustments are in the present and the next moment, inherently out of sync. Without unified coordinate calibration, 3D reference trajectory generation, and time delay compensation mechanisms, it is difficult to achieve accurate monitoring, timely correction, and reliable alarms for the welding robot's working status. The hand-eye calibration, 3D reconstruction, NURBS trajectory fitting, MPC real-time correction, and monitoring software platform proposed in the appendix reflect the urgent need for existing systems to evolve from single recognition functions to full-process closed-loop monitoring.

[0006] Therefore, it is necessary to propose a machine vision-based method and system for monitoring the operation of welding robots to solve at least one of the following problems in the existing technology: unstable extraction of weld features under strong interference environment, difficulty in obtaining weld spatial information and welding gun posture in a unified manner, lack of continuous monitoring of welding working status, time lag mismatch between visual feedback and robot movement, and difficulty in timely alarm and traceability of abnormal working conditions. This will improve the operational stability, monitoring accuracy and / or welding quality consistency of welding robots. Summary of the Invention

[0007] In view of this, the purpose of this invention is to propose a machine vision-based welding robot operation monitoring method and system that can improve the operational stability, monitoring accuracy and / or welding quality consistency of welding robots.

[0008] To achieve the above-mentioned technical objectives, the technical solution adopted by this invention is as follows: A machine vision-based method for monitoring the operation of a welding robot is applied to a welding robot vision monitoring platform. The platform includes an industrial robot, an industrial control computer, a human-machine interface terminal (HMI), and a welding head, machine vision sensors, and a robot controller deployed on the industrial robot. The machine vision sensors are used for global monitoring of the welding area. The industrial control computer is communicatively connected to the robot controller for operation control and information exchange with the industrial robot, machine vision sensors, and welding head. The HMI is communicatively connected to the industrial control computer and the robot controller for work information exchange. The operation monitoring method includes: S1. Obtain the planned welding trajectory, welding process parameters and monitoring thresholds of the workpiece to be welded, and establish a monitoring object set for the welding task; S2. Perform camera intrinsic parameter calibration and distortion correction on the machine vision sensor, perform hand-eye calibration between the machine vision sensor and the industrial robot base coordinate system, and calibrate the center point of the welding tool to establish a unified coordinate transformation relationship between the vision coordinate system, the robot coordinate system and the welding tool coordinate system. S3. Collect image sequences of the welding area before and during welding, and simultaneously collect the pose information of the industrial robot and welding process parameter information. S4. Preprocess the acquired image sequence and construct a dynamic region of interest based on the weld feature position of the previous moment to obtain the candidate welding monitoring area at the current moment. S5. Perform weld feature extraction within the candidate welding monitoring area to obtain two-dimensional weld feature and depth information, and calculate the working monitoring parameters of the welding torch relative to the weld. S6. Based on the extracted two-dimensional weld features and depth information, construct a three-dimensional point cloud of the welding area, register and fuse multiple frames of point clouds, and perform curve fitting on the weld space trajectory to generate a three-dimensional reference trajectory of the weld and the corresponding theoretical attitude field of the welding gun. S7. The current weld characteristics, work monitoring parameters, weld three-dimensional reference trajectory, industrial robot actual pose and welding process parameters are fused together to determine the current welding working status of the industrial robot.

[0009] As one possible implementation, the industrial robot described in this solution is a six-axis industrial robot; the machine vision sensor is an active vision sensor, which is a line structure light sensor. The line structure light sensor is installed on the flange structure at the end of the industrial robot and is rigidly connected to the welding head to form an eye-on-hand visual monitoring structure.

[0010] As a possible implementation, further, in step S2 of this scheme, the camera intrinsic parameter calibration adopts the Zhang Zhengyou calibration method, and the hand-eye calibration adopts the matrix solution method based on standard sphere multi-pose sampling to obtain the rotation matrix and translation vector of the machine vision sensor relative to the end flange of the industrial robot, and further combines the calibration results of the welding tool center point to establish a unified coordinate transformation model.

[0011] As a possible implementation, further, in step S4 of this scheme, the preprocessing of the image sequence includes: distortion correction, grayscale normalization, anisotropic diffusion filtering, morphological opening and closing operations, and background suppression; the dynamic region of interest is updated according to the weld center position, bevel edge position, or structured light center position of the previous frame.

[0012] As a possible implementation, further, in step S5 of this scheme, the weld feature extraction is achieved by combining a lightweight semantic segmentation network with sub-pixel localization. First, the lightweight semantic segmentation network is used to obtain the weld candidate region, and then the gray-scale centroid method or the edge normal intersection method is used to calculate the coordinates of the weld center line or the coordinates of the feature points.

[0013] As one possible implementation, further, in step S5 of this solution, after performing weld feature extraction, at least one feature among weld centerline, groove edge, structured light stripe, molten pool outline and welding torch projection is obtained; The monitoring parameters are at least one of the following: lateral deviation, height deviation, and attitude deviation of the welding torch relative to the weld.

[0014] As a possible implementation, further, in step S6 of this scheme, when registering and fusing multi-frame point clouds, a combination of iterative nearest point registration and registration based on normal vector features is adopted; when fitting the weld space trajectory, non-uniform rational B-spline curve fitting is adopted, and the theoretical attitude field of the welding gun is generated based on the tangent vector and normal vector of the fitted curve. In S7, the welding working state includes at least the normal working state, the state to be corrected state, and the alarm state.

[0015] As one possible implementation, further, in step S7 of this solution, the working monitoring parameters include at least lateral deviation, height deviation, posture deviation, weld continuity, and visual interference index; When the lateral deviation, height deviation, and posture deviation are all within the preset first-level threshold range, and the weld continuity is not lower than the continuity threshold and the visual interference index is not higher than the interference threshold, it is judged as a normal working state. When at least one monitoring parameter exceeds the first-level threshold but does not exceed the second-level threshold, it is determined to be in a state requiring correction. An alarm is triggered when at least one monitoring parameter exceeds the secondary threshold or when no valid weld features are extracted for multiple consecutive frames.

[0016] As a preferred implementation method, this solution further includes: S8. When the judgment result is a state to be corrected, the work monitoring parameters are compensated and predicted based on the visual feedback delay and the robot motion response delay, and the robot end correction command is output; when the judgment result is an alarm state, the alarm information is output and the abnormal image, abnormal location and process parameters are recorded.

[0017] As a preferred implementation method, preferably, in step S8 of this scheme, the visual feedback delay includes image acquisition delay, image processing delay and communication delay, and the robot motion response delay includes controller processing delay and actuator inertial response delay; a discrete state prediction model is established based on the visual feedback delay and robot motion response delay, and a robot end effector correction command is output after forward prediction of the current working monitoring parameters.

[0018] Based on the above, this solution also proposes a machine vision-based welding robot work monitoring system, which includes: The task initialization module is used to read the theoretical welding trajectory, welding process parameters and monitoring thresholds of the workpiece to be welded, and to establish a set of monitoring objects for the welding task. The calibration modeling module is used to perform camera intrinsic parameter calibration, hand-eye calibration, and welding tool center point calibration to establish a unified coordinate transformation relationship; The vision acquisition module is used to acquire image sequences before and during welding, and simultaneously acquire the pose information of the industrial robot and welding process parameter information. The image preprocessing module is used to preprocess image sequences and construct dynamic regions of interest. The feature extraction module is used to extract at least one feature from the weld centerline, groove edge, structured light stripes, molten pool contour, and welding torch projection, and to calculate the working monitoring parameters. The 3D reconstruction module is used to construct a 3D point cloud of the welding area and generate a 3D reference trajectory of the weld and a theoretical attitude field of the welding torch. The status assessment module is used to integrate weld features, work monitoring parameters, weld three-dimensional reference trajectory, actual pose of industrial robot and welding process parameters, and determine the current welding work status. The compensation output module is used to output robot end effector correction commands when the current welding working state is in the state to be corrected. The alarm recording module is used to output alarm information and record abnormal images, abnormal locations and process parameters when the current welding working state is in an alarm state. The vision acquisition module includes a line structured light vision sensor and a narrow band filter assembly. The line structured light vision sensor is installed at the end flange of the industrial robot and rigidly connected to the welding head. The status assessment module is used to output at least one status result among normal operation status, pending correction status, alarm status, visual failure status and shutdown re-inspection status, and display the status result on the human-computer interaction terminal. The system also includes a monitoring software platform, which displays weld images, tracks deviation curves, welding process parameters, robot status information, abnormal alarm information, and process reports. This corresponds to the HMI, reports, and status monitoring content disclosed in the appendix.

[0019] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art: This solution constructs a visual monitoring platform consisting of an industrial robot, welding head, machine vision sensors, industrial computer, and controller. By combining camera intrinsic parameter calibration, hand-eye calibration, and tool center point calibration, a unified mapping relationship is established between visual coordinates, robot coordinates, and welding tool coordinates. This enables precise correspondence between the weld position and geometric features detected by vision and the robot's end effector pose, solving the problem of "seeing but not aligning" in existing welding systems and improving the welding robot's perception accuracy of weld position and trajectory execution accuracy.

[0020] This solution also preprocesses welding images, dynamically locks regions of interest, and extracts weld features. By combining lightweight deep learning segmentation with traditional sub-pixel localization methods, it can stably extract key features such as weld centerline, bevel edge, or structured light stripes under complex interference environments such as strong arc light, spatter, and high reflectivity. This significantly enhances the system's adaptability and robustness to harsh working conditions and solves the problems of weld visual features being easily affected by environmental interference and poor stability of continuous detection in existing technologies.

[0021] In addition, this solution combines two-dimensional weld features with depth information to form a three-dimensional point cloud, and uses point cloud registration fusion and NURBS curve fitting to generate a three-dimensional reference trajectory of the weld and a theoretical attitude field of the welding torch. This can transform discrete and local visual inspection results into a continuous, smooth, and executable spatial welding trajectory, and realize the integrated planning of welding torch position and attitude. This solves the problem that existing technologies cannot directly obtain the three-dimensional trajectory of complex spatial welds and the optimal attitude of the welding torch from two-dimensional images, thereby improving the trajectory planning capability and welding consistency of complex curved welds and various joint types.

[0022] Finally, this solution determines the working status by integrating weld features, deviation parameters, actual robot pose, and welding process parameters. In the state to be corrected, visual feedback time delay compensation and model predictive control are introduced to achieve online correction. In the alarm state, abnormal alarms, data recording, and process traceability are performed. This solution can effectively overcome the tracking deviation problems caused by assembly errors, thermal deformation, and asynchronous visual sampling and robot response, improve the dynamic adaptive capability, closed-loop response capability, and quality consistency of the welding process, and facilitate subsequent process analysis and quality traceability.

[0023] In addition to the above, this solution can also centrally display and manage weld images, deviation curves, welding parameters, robot status and alarm information through a monitoring software platform, realizing integrated collaboration of weld recognition, trajectory generation, real-time correction and process monitoring, thereby shortening the changeover and debugging cycle, and improving the flexible production capacity and engineering application value under multi-variety and small-batch welding tasks.

[0024] In summary, this solution organically integrates multi-sensor unified calibration, robust visual extraction of weld seams, 3D reconstruction and attitude planning, working status assessment, time delay compensation correction, and process monitoring into a single system, constructing a closed-loop monitoring mechanism of "perception-reconstruction-assessment-control-monitoring." This mechanism not only stably acquires weld seam features and generates high-precision spatial trajectories under strong interference welding environments, but also dynamically corrects deviations and provides early warnings of anomalies for the welding robot based on real-time deviations. This effectively improves weld seam tracking accuracy, process stability, welding quality consistency, and the system's flexible application capabilities. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is one of the simplified implementation flowcharts of the monitoring method in this scheme; Figure 2 This is the second simplified implementation flowchart of the monitoring method in this scheme; Figure 3 This is a simplified connection diagram of the unit modules of the monitoring system in this solution. Detailed Implementation

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] like Figure 1As shown in this embodiment, a welding robot work monitoring method based on machine vision is applied to a welding robot vision monitoring platform. The platform includes an industrial robot, an industrial control computer, a human-machine interface terminal, and a welding head, machine vision sensors, and a robot controller deployed on the industrial robot. The machine vision sensors are used for global monitoring of the welding area. The industrial control computer is communicatively connected to the robot controller for work control and information interaction with the industrial robot, machine vision sensors, and welding head. The human-machine interface terminal is communicatively connected to the industrial control computer and the robot controller for work information interaction. The work monitoring method includes: S1. Obtain the planned welding trajectory, welding process parameters and monitoring thresholds of the workpiece to be welded, and establish a monitoring object set for the welding task; S2. Perform camera intrinsic parameter calibration and distortion correction on the machine vision sensor, perform hand-eye calibration between the machine vision sensor and the industrial robot base coordinate system, and calibrate the center point of the welding tool to establish a unified coordinate transformation relationship between the vision coordinate system, the robot coordinate system and the welding tool coordinate system. S3. Collect image sequences of the welding area before and during welding, and simultaneously collect the pose information of the industrial robot and welding process parameter information. S4. Preprocess the acquired image sequence and construct a dynamic region of interest based on the weld feature position of the previous moment to obtain the candidate welding monitoring area at the current moment. S5. Perform weld feature extraction within the candidate welding monitoring area to obtain two-dimensional weld feature and depth information, and calculate the working monitoring parameters of the welding torch relative to the weld. S6. Based on the extracted two-dimensional weld features and depth information, construct a three-dimensional point cloud of the welding area, register and fuse multiple frames of point clouds, and perform curve fitting on the weld space trajectory to generate a three-dimensional reference trajectory of the weld and the corresponding theoretical attitude field of the welding gun. S7. The current weld characteristics, work monitoring parameters, weld three-dimensional reference trajectory, industrial robot actual pose and welding process parameters are fused together to determine the current welding working status of the industrial robot.

[0029] As one possible implementation, the industrial robot described in this solution is a six-axis industrial robot; the machine vision sensor is an active vision sensor, which is a line structure light sensor. The line structure light sensor is installed on the flange structure at the end of the industrial robot and is rigidly connected to the welding head to form an eye-on-hand visual monitoring structure.

[0030] As an example, in this solution, the welding robot vision monitoring platform may include the following main unit module roles: (1) Six-axis industrial robot; (2) Laser welding head or arc welding gun, which serves as the welding head part; (3) Active machine vision sensor, preferably a line structured light vision sensor; (4) Industrial PC; (5) Robot controller; (6) Human-Computer Interaction Terminal (HMI); (7) The process database and process data storage module are deployed on the server and are connected to the industrial control computer, human-machine interface terminal (HMI) and / or robot controller.

[0031] The vision sensor is installed at the end flange of the robot and maintains a fixed relative pose with the welding head; the industrial control computer is responsible for image acquisition, image processing, 3D reconstruction, status assessment, and trajectory correction calculation; the robot controller is responsible for receiving correction commands and driving the servo joints to move; the HMI is used to display weld images, deviation curves, welding parameters, alarm status, and process reports.

[0032] In the above example scheme, the machine vision sensor is a sensor group, which includes the necessary auxiliary components to realize its function. Similarly, the other unit modules are as follows, which are all common existing hardware solutions, and will not be described in detail here.

[0033] In step S1 of this scheme, the planned welding trajectory, welding process parameters and monitoring thresholds of the workpiece to be welded can be obtained by reading the welding task information, and a monitoring object set for the welding task can be established.

[0034] As an example, during the task initialization phase, the theoretical weld model, welding speed, welding power, allowable deviation threshold, and alarm threshold can be established by obtaining the process documents of the current welding task, and a set of monitoring objects can be formed.

[0035] The task initialization result can be represented as follows:

[0036] in, Theoretical geometric information of the workpiece to be welded; This is a set of welding process parameters; It is a collection of vision sensors and robot hardware configurations that includes planned welding trajectory information; This is a set of monitoring rules and thresholds, which contains relevant information about the monitoring thresholds; This is a set of alarm and shutdown policies.

[0037] Furthermore, the monitoring threshold vector can be defined as follows:

[0038] in, This refers to the allowable threshold for lateral deviation. Allowable threshold for height deviation; This refers to the allowable threshold for attitude deviation. This is the threshold for weld continuity. This is the threshold for the visual interference index.

[0039] Based on the above, as a possible implementation, further, in step S2 of this solution, the camera intrinsic parameter calibration adopts the Zhang Zhengyou calibration method, and the hand-eye calibration adopts the matrix solution method based on standard sphere multi-pose sampling to obtain the rotation matrix and translation vector of the machine vision sensor relative to the end flange of the industrial robot, and further combines the calibration results of the welding tool center point to establish a unified coordinate transformation model.

[0040] As an example, in step S2 of this scheme, Zhang Zhengyou's calibration method is used for camera intrinsic parameter calibration, and a matrix solution method based on a standard sphere (AX=XB) is used for hand-eye relationship calibration. Further TCP precision calibration and global error compensation are then performed. This method eliminates spatial transformation errors between the vision sensor, the robot end flange, and the welding torch TCP, enabling weld features in the image to be mapped seamlessly to the robot's motion control space.

[0041] Specifically, step S2 may include the following sub-steps: 2.1 Camera Imaging Model Establishment A pinhole imaging model is established, defined as follows:

[0042] in, Scale factor; These are pixel coordinates; The coordinates of a point in the world coordinate system; This is the camera intrinsic parameter matrix; This is the rotation matrix from the world coordinate system to the camera coordinate system; This is the translation vector from the world coordinate system to the camera coordinate system.

[0043] The camera intrinsic parameter matrix is ​​defined as follows:

[0044] in, , Equivalent focal length in pixel coordinate system , All are the coordinates of the principal point.

[0045] 2.2 Distortion Correction Considering radial and tangential distortion, normalize the image points. Image points after distortion correction The relationship is as follows:

[0046]

[0047]

[0048] in, , Radial distortion coefficient; , The tangential distortion coefficient; This is the distance from the normalized image point to the center of the optical axis.

[0049] The above definition logic is based on a real lens, which is a non-ideal pinhole model with non-linear offset in the edge region. Therefore, radial and tangential correction terms need to be superimposed on the ideal projection to restore geometric realism.

[0050] 2.3 Hand-eye calibration model Let the robot base coordinate system be... The coordinate system of the end flange is The camera coordinate system is The calibration sphere or calibration body coordinate system is Define the hand-eye transformation matrix as follows:

[0051] For the robot in two sets of postures , The collected data includes:

[0052] After tidying up both sides:

[0053] in:

[0054]

[0055] The above is the form AX=XB for hand-eye calibration. Its derivation is essentially this: the same calibration object remains unchanged in the world; only the "hand" and "eye" move independently. Therefore, the relative motion between the two segments must be mediated by an unknown hand-eye relationship. Connect them.

[0056] in, For the first The pose of the end flange relative to the robot base during the next sampling; For the first The pose of the calibration body relative to the camera during the next sampling; The relative motion matrix of the flange; The relative motion matrix of the calibration body; Let be the hand-eye transformation matrix to be determined.

[0057] 2.4 TCP Labeling and Unified Mapping Let the transformation matrix of the welding torch TCP (i.e., the welding head) relative to the end flange be as follows:

[0058] Then any point in the camera coordinate system Mapped to the robot's base coordinate system, it can be represented as:

[0059] The base position of the welding torch TCP is:

[0060] Therefore, the weld points acquired by the camera TCP position of welding torch They are uniformly expressed in the robot base coordinate system, providing a direct basis for subsequent deviation calculations.

[0061] In step S3 of this scheme, images, robot poses and process data are stably acquired during the welding process, so that the multi-source data are strictly aligned on the time axis, providing synchronous input for subsequent feature extraction and deviation calculation.

[0062] As an example, step S3 of this solution may include the following: 3.1 Definition of Multi-Source Data At each sampling time The system collects the following data:

[0063] in, For the first Frame of the original image; This refers to the robot's joint angle vectors. Position of the end flange; This is a vector of process parameters; To standardize timestamps.

[0064] The process parameter vector can be defined as:

[0065] in: This refers to the laser power or equivalent thermal input parameter. For welding speed; This refers to the welding current. This refers to the welding voltage; This refers to the wire feeding speed.

[0066] 3.2 Time Synchronization and Interpolation Since the visual acquisition frequency, robot control frequency, and process sampling frequency may differ, a unified discrete-time grid is established:

[0067] in, The start time of the mission; To standardize the sampling period, first-order linear interpolation is used for asynchronous signals, defined as follows:

[0068] in, For any non-visual data to be synchronized; , To meet Adjacent sampling times; To synchronize to a unified time The estimated values ​​can then be used to form strictly aligned data packets.

[0069] 3.3 Initial Delay Estimation Define total delay for:

[0070] in, For image exposure and acquisition delay; Image processing latency; For communication delay; This is to account for the response delay of the implementing agency.

[0071] As a possible implementation method, the preprocessing of the image sequence in step S4 of this scheme includes: distortion correction, grayscale normalization, anisotropic diffusion filtering, morphological opening and closing operations, and background suppression; the dynamic region of interest is updated according to the weld center position, bevel edge position, or structured light center position of the previous frame.

[0072] To reliably preserve effective information such as weld edges and structured light fringes from the original welding image under conditions of strong arc light, spatter, high reflectivity, and smoke interference, and to reduce the subsequent search area, thereby decreasing computational load and false detection rate, this scheme employs anisotropic diffusion filtering and morphological operations for preprocessing, and designs a ROI dynamic tracking algorithm based on the feature positions of the previous frame.

[0073] As an example, step S4 of this solution may include the following: 4.1 Denoising and Edge Preservation The original image was processed using an anisotropic diffusion model. Its continuous form is:

[0074] in, The grayscale function is used to represent the image's grayscale value. For diffusion iteration time; Image gradient; It is a function of the diffusion coefficient; ( ) is the divergence operator.

[0075] Preferably, it can also be defined as follows:

[0076] in, This is an edge-sensitive parameter. Its derivation logic is as follows: in a flat region, Small diffusion coefficient, large diffusion coefficient, enhances noise reduction; in the edge region "Large and small diffusion coefficient, thus avoiding the weld edge being smeared into a pot of mortar."

[0077] 4.2 Morphological Enhancement For the diffused image The opening and closing operations are defined as follows:

[0078]

[0079] in, For erosion calculation; For expansion operation; B is a structuring element.

[0080] Opening operations are used to remove isolated bright noise, while closing operations are used to fill small cracks, thereby enhancing the profile of continuous weld seams.

[0081] 4.3 Dynamic ROI Locking Let the center point of the weld extracted from the previous frame be... The region of interest for the current frame is then defined as:

[0082] in, For the first ROI of the frame; , Define the width and height of the ROI.

[0083] To balance stability and adaptability, we can set:

[0084]

[0085] in, , Minimum window size; , This is the scaling factor; , This represents the standard deviation of the horizontal and vertical distribution of feature points in the previous frame.

[0086] Step S4 allows the ROI to be dynamically adjusted as the weld position slowly drifts, without sudden abnormal jumps due to single-frame noise.

[0087] As a possible implementation, further, in step S5 of this scheme, the weld feature extraction is achieved by combining a lightweight semantic segmentation network with sub-pixel localization. First, the lightweight semantic segmentation network is used to obtain the weld candidate region, and then the gray-scale centroid method or the edge normal intersection method is used to calculate the coordinates of the weld center line or the coordinates of the feature points.

[0088] In step S5 of this scheme, after performing weld feature extraction, at least one feature is obtained from the weld centerline, bevel edge, structured light stripes, molten pool outline, and welding torch projection; the working monitoring parameter is at least one of the lateral deviation, height deviation, and attitude deviation of the welding torch relative to the weld.

[0089] As an example, step S5 of this solution may include the following: 5.1 Semantic segmentation yields candidate weld regions The ROI image is fed into a lightweight U-Net network to obtain a weld probability map, the function of which is defined as follows:

[0090] in, For pixels The probability that it belongs to the weld area; For parameters A lightweight U-Net model.

[0091] During model training, a weighted form of binary cross-entropy and Dice loss can be used, defined as follows:

[0092] in, For binary cross-entropy loss; For Dice's loss; , This is the loss weight. The purpose of this segmentation step is to roughly determine the location of the weld seam, providing a technical basis for subsequent sub-pixel positioning.

[0093] 5.2 Extracting the center line using the grayscale centroid method For each row or column of effective grayscale distribution, the center position is calculated using the grayscale centroid method:

[0094] in, Let v be the x-coordinate of the weld center at the y-coordinate. ,v) represents the preprocessed pixel grayscale or probability weighted value. If structured light center fringe detection is used, the fringe center can also be obtained using the above formula on the fringe cross-section.

[0095] 5.3 Extraction of bevel edge and center point Let the left and right edge points be respectively Then the center point of the weld section can be represented as:

[0096] in, This is the edge point of the left bevel; This is the edge point of the right bevel; This is the center point of the weld section.

[0097] When using the edge normal intersection method, the local curves of the left and right edges can be fitted first to obtain their normal equations, and then the intersection of the normals or the midpoint of the minimum distance can be found to enhance the adaptability to asymmetric bevels.

[0098] 5.4 Calculate working monitoring parameters Based on the coordinate mapping relationship of S2, the image features are mapped to the base coordinate system to obtain the current weld reference point. and the TCP position of the welding torch .

[0099] (1) Lateral deviation Let the local normal transverse basis vector of the weld be... Then the lateral deviation is:

[0100] in, This is the lateral deviation; It is a local horizontal unit vector.

[0101] (2) Height deviation Let the local surface normal vector of the weld be... The height deviation is:

[0102] in, For height deviation; It is the local surface normal unit vector.

[0103] (3) Attitude deviation Let the unit vector of the welding torch axis be... The desired incident direction is Then the attitude deviation is:

[0104] in, This is the attitude deviation angle; This is the actual axial unit vector of the welding torch; This is the theoretical axial unit vector of the welding torch.

[0105] (4) Weld continuity The relevant function is defined as follows:

[0106] in, Weld continuity; This represents the number of valid feature points detected in this frame. Theoretically, the number of feature points should be detected.

[0107] (5) Visual interference index Define the overexposed area ratio, the blocked area ratio, and the splatter bright spot density as follows: , , The visual interference index is:

[0108] in, Visual interference index; , , The weights are for each interference component.

[0109] In step S6 of this scheme, when registering and fusing multi-frame point clouds, a combination of iterative nearest-point registration and registration based on normal vector features is adopted; when fitting the weld space trajectory, non-uniform rational B-spline curve fitting is adopted, and the theoretical attitude field of the welding gun is generated based on the tangent vector and normal vector of the fitted curve. As an example, step S6 of this solution may include the following: 6.1 Conversion of 2D Feature Points to 3D Point Clouds Image feature points If the depth is obtained by structured light triangulation or a depth module Then the three-dimensional points in the camera coordinate system are:

[0110] in, A 3D point in the camera coordinate system; The inverse of the camera intrinsic parameter matrix; This corresponds to the depth value.

[0111] Further transformation to the base coordinate system based on the coordinate mapping relationship of S2:

[0112] This yields the point cloud of the current frame. .

[0113] 6.2 Multi-frame point cloud ICP registration Let the point cloud of the current frame be Historical reference point cloud is Then through rigid body transformation To achieve optimal alignment between the two, the optimization objective is:

[0114] in. It is a rotation matrix; It is a translation vector; The corresponding nearest neighbor point in the reference point cloud; This represents the number of points involved in registration for the current frame.

[0115] This step is based on the fact that the same weld seam is locally scanned in multiple consecutive frames. The only difference between different frames is the rigid body pose. Therefore, the overall consistent three-dimensional geometry can be recovered by minimizing the square distance between corresponding points.

[0116] 6.3 NURBS Trajectory Fitting For the fused weld point set For NURBS fitting, the function is defined as follows:

[0117] in, This represents the three-dimensional trajectory curve of the weld. For p-th degree B-spline basis functions; For the first i Weights of each control point; For the first i There are 1 control point; u is the curve parameter.

[0118] The rationale for using NURBS in this step is that discrete point clouds are noisy strings of points, while robots require differentiable, derivativeable, and interpolable continuous curves; therefore, NURBS is well-suited for transforming coarse point clouds into smooth trajectories.

[0119] 6.4 Solving for Tangent Vector, Normal Vector, and Attitude Field The trajectory tangent vector is defined as:

[0120] in, The first derivative of the trajectory; It is the unit tangent vector.

[0121] The local surface normal vector of the weld can be obtained by fitting the local neighborhood plane of the point cloud. The covariance matrix of the neighborhood point set is defined as the following function:

[0122] The eigenvector corresponding to the smallest eigenvalue is denoted as . That is, the local surface normal vector. For neighboring points; is the mean of the neighborhood points; m is the number of neighborhood points; This is the local surface normal vector.

[0123] Further define the binormal vector:

[0124] If the welding torch is expected to be at an incident angle relative to the surface normal... Upon entry, the theoretical axial direction of the welding torch is:

[0125] Based on this, an attitude rotation matrix can be formed:

[0126] Then convert it to Euler angles. The six-dimensional theoretical motion command is obtained:

[0127] The output of this step is as follows: 3D point cloud model of weld ; Three-dimensional reference trajectory Theoretical attitude field or six-dimensional instructions .

[0128] In this scheme S7, the welding working state includes at least the normal working state, the state to be corrected state, and the alarm state.

[0129] As one possible implementation, further, in step S7 of this solution, the working monitoring parameters include at least lateral deviation, height deviation, posture deviation, weld continuity, and visual interference index; When the lateral deviation, height deviation, and posture deviation are all within the preset first-level threshold range, and the weld continuity is not lower than the continuity threshold and the visual interference index is not higher than the interference threshold, it is judged as a normal working state. When at least one monitoring parameter exceeds the first-level threshold but does not exceed the second-level threshold, it is determined to be in a state requiring correction. An alarm is triggered when at least one monitoring parameter exceeds the secondary threshold or when no valid weld features are extracted for multiple consecutive frames.

[0130] The main technical problem addressed in step S7 of this solution is to integrate visual features, deviation parameters, three-dimensional reference trajectory, robot actual pose, and process data into a unified state criterion to distinguish different working states such as normal operation, waiting for correction, and alarm.

[0131] As an example, step S7 of this solution includes the following: 7.1 Construction of State Feature Vectors Construct the working state feature vector:

[0132] in, This is the lateral deviation; For height deviation; This is for attitude deviation; For weld continuity; Visual interference index; For welding speed; This refers to welding power or heat input parameters.

[0133] 7.2 Normalized Risk Assessment Indicators To unify parameters with different dimensions, a normalized risk index is defined:

[0134] in, As a comprehensive risk indicator; For each weight coefficient, satisfying .

[0135] This step converts deviations, continuity gaps, and visual interference into "relative threshold exceedances," and then sums them by importance to obtain a unified risk scale.

[0136] 7.3 State Determination Rules The status determination can be performed according to the following rules: Normal operating status

[0137] pending correction At least one parameter exceeds the first-level threshold, but does not reach the shutdown threshold, and continuous effective features can still be extracted stably.

[0138] Alarm status An alarm state is determined when any of the following conditions are met:

[0139] or

[0140] Or, effective weld features cannot be extracted in M ​​consecutive frames.

[0141] in, is the secondary threshold; M is the threshold for the number of consecutive failure frames.

[0142] Based on the above scheme, combined with Figure 2 As shown, as a preferred embodiment, this solution, in addition to the steps S1-S7 described above, may further include: S8. When the judgment result is a state to be corrected, the work monitoring parameters are compensated and predicted based on the visual feedback delay and the robot motion response delay, and the robot end correction command is output; when the judgment result is an alarm state, the alarm information is output and the abnormal image, abnormal location and process parameters are recorded.

[0143] As a preferred implementation method, preferably, in step S8 of this scheme, the visual feedback delay includes image acquisition delay, image processing delay and communication delay, and the robot motion response delay includes controller processing delay and actuator inertial response delay; a discrete state prediction model is established based on the visual feedback delay and robot motion response delay, and a robot end effector correction command is output after forward prediction of the current working monitoring parameters.

[0144] In step S8 of this solution, under the condition that there are time delays in visual acquisition, computational processing, communication and robot execution, the detected deviation is converted into an effective correction command, and alarm, shutdown and full process traceability are completed in abnormal situations. By establishing a discrete state-space model of the system that includes image processing delay, communication delay and robot first-order inertial link, rolling time domain optimization based on model predictive control is adopted to achieve real-time deviation correction. At the same time, status monitoring, out-of-tolerance alarm, data recording and process report generation are realized in the software platform.

[0145] As an example, step S8 of this solution includes: 8.1 Establishing a Discrete State-Space Model with Time Delay Define the error state vector:

[0146] Control input is defined as:

[0147] in, This represents the horizontal adjustment amount for the current cycle; This represents the current cycle height correction amount.

[0148] Considering time delay Establish a discrete model:

[0149]

[0150] in, This is the state transition matrix; The input matrix; Here is the perturbation matrix; External disturbances characterize assembly errors, thermal deformation, etc. For observable output; This is the output matrix.

[0151] 8.2 MPC Optimization Objective Construction In the prediction time domain and control time domain Internal structural performance indicators:

[0152] in, Optimize the target for MPC; This is a prediction of the state at time k+i in the future. The desired reference state is typically taken as the zero-deviation state; This is the state error weight matrix; To control the incremental weight matrix; To control the input increment.

[0153] By solving this optimization problem, the optimal control sequence is obtained:

[0154] Then only the first current control value is set. Send the data to the robot controller to implement the rolling optimization concept.

[0155] 8.3 Modification Instruction Generation If the current state is Then, the welding torch TCP corrected pose is generated:

[0156] in, This represents the current TCP pose. For the reason The resulting pose correction matrix; This is the corrected TCP pose.

[0157] If the current state If this happens, the output of welding motion correction commands will be prohibited, and an emergency stop or shutdown re-inspection process will be triggered.

[0158] 8.4 Process monitoring and alarm recording Create process log entries:

[0159] in, For timestamps; For raw or labeled images; This is the state feature vector; As a comprehensive risk indicator; For status labels; This represents the actual TCP pose. These are process parameters.

[0160] when At that time, the system will execute one of the following: 1. Record abnormal images; 2. Record the location and type of the anomaly; 3. Record process parameters and robot status; 4. Generate human-machine interface alarms; 5. Generate process reports.

[0161] Combination Figure 3 As shown above, this solution also proposes a machine vision-based welding robot work monitoring system, which includes: The task initialization module is used to read the theoretical welding trajectory, welding process parameters and monitoring thresholds of the workpiece to be welded, and to establish a set of monitoring objects for the welding task. The calibration modeling module is used to perform camera intrinsic parameter calibration, hand-eye calibration, and welding tool center point calibration to establish a unified coordinate transformation relationship; The vision acquisition module is used to acquire image sequences before and during welding, and simultaneously acquire the pose information of the industrial robot and welding process parameter information. The image preprocessing module is used to preprocess image sequences and construct dynamic regions of interest. The feature extraction module is used to extract at least one feature from the weld centerline, groove edge, structured light stripes, molten pool contour, and welding torch projection, and to calculate the working monitoring parameters. The 3D reconstruction module is used to construct a 3D point cloud of the welding area and generate a 3D reference trajectory of the weld and a theoretical attitude field of the welding torch. The status assessment module is used to integrate weld features, work monitoring parameters, weld three-dimensional reference trajectory, actual pose of industrial robot and welding process parameters, and determine the current welding work status. The compensation output module is used to output robot end effector correction commands when the current welding working state is in the state to be corrected. The alarm recording module is used to output alarm information and record abnormal images, abnormal locations and process parameters when the current welding working state is in an alarm state. The vision acquisition module includes a line structured light vision sensor and a narrow band filter assembly. The line structured light vision sensor is installed at the end flange of the industrial robot and rigidly connected to the welding head. The status assessment module is used to output at least one status result among normal operation status, pending correction status, alarm status, visual failure status and shutdown re-inspection status, and display the status result on the human-computer interaction terminal. The system also includes a monitoring software platform, which displays weld images, tracks deviation curves, welding process parameters, robot status information, abnormal alarm information, and process reports. This corresponds to the HMI, reports, and status monitoring content disclosed in the appendix.

[0162] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0163] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A machine vision-based method for monitoring the operation of a welding robot, applied to a welding robot vision monitoring platform, the platform comprising an industrial robot, an industrial control computer, a human-machine interface terminal, and a welding head, machine vision sensors, and a robot controller deployed on the industrial robot; the machine vision sensors are used for global monitoring of the welding area; the industrial control computer is communicatively connected to the robot controller for operation control and information interaction with the industrial robot, machine vision sensors, and welding head; the human-machine interface terminal is communicatively connected to the industrial control computer and the robot controller for work information interaction; characterized in that... The work monitoring method includes: S1. Obtain the planned welding trajectory, welding process parameters and monitoring thresholds of the workpiece to be welded, and establish a monitoring object set for the welding task; S2. Perform camera intrinsic parameter calibration and distortion correction on the machine vision sensor, perform hand-eye calibration between the machine vision sensor and the industrial robot base coordinate system, and calibrate the center point of the welding tool to establish a unified coordinate transformation relationship between the vision coordinate system, the robot coordinate system and the welding tool coordinate system. S3. Collect image sequences of the welding area before and during welding, and simultaneously collect the pose information of the industrial robot and welding process parameter information. S4. Preprocess the acquired image sequence and construct a dynamic region of interest based on the weld feature position of the previous moment to obtain the candidate welding monitoring area at the current moment. S5. Perform weld feature extraction within the candidate welding monitoring area to obtain two-dimensional weld feature and depth information, and calculate the working monitoring parameters of the welding torch relative to the weld. S6. Based on the extracted two-dimensional weld features and depth information, construct a three-dimensional point cloud of the welding area, register and fuse multiple frames of point clouds, and perform curve fitting on the weld space trajectory to generate a three-dimensional reference trajectory of the weld and the corresponding theoretical attitude field of the welding gun. S7. The current weld characteristics, work monitoring parameters, weld three-dimensional reference trajectory, industrial robot actual pose and welding process parameters are fused together to determine the current welding working status of the industrial robot.

2. The machine vision-based welding robot operation monitoring method as described in claim 1, characterized in that, The industrial robot is a six-axis industrial robot; the machine vision sensor is an active vision sensor, which is a line structure light sensor. The line structure light sensor is installed on the flange structure at the end of the industrial robot and is rigidly connected to the welding head to form an eye-on-hand visual monitoring structure.

3. The machine vision-based welding robot operation monitoring method as described in claim 1, characterized in that, In step S2, the camera intrinsic parameter calibration adopts the Zhang Zhengyou calibration method, and the hand-eye calibration adopts the matrix solution method based on standard sphere multi-pose sampling to obtain the rotation matrix and translation vector of the machine vision sensor relative to the end flange of the industrial robot, and further combines the calibration results of the welding tool center point to establish a unified coordinate transformation model.

4. The machine vision-based welding robot operation monitoring method as described in claim 1, characterized in that, In step S4, the preprocessing of the image sequence includes: distortion correction, grayscale normalization, anisotropic diffusion filtering, morphological opening and closing operations, and background suppression; the dynamic region of interest is updated based on the weld center position, bevel edge position, or structured light center position of the previous frame.

5. The machine vision-based welding robot operation monitoring method as described in claim 1, characterized in that, In step S5, weld feature extraction is achieved by combining a lightweight semantic segmentation network with sub-pixel localization. First, the lightweight semantic segmentation network is used to obtain the weld candidate region, and then the gray-scale centroid method or the edge normal intersection method is used to calculate the coordinates of the weld center line or the coordinates of the feature points. In step S5, after performing weld feature extraction, at least one feature among the weld centerline, bevel edge, structured light stripes, molten pool outline, and welding torch projection is obtained. The monitoring parameters are at least one of the following: lateral deviation, height deviation, and attitude deviation of the welding torch relative to the weld.

6. The machine vision-based welding robot operation monitoring method as described in claim 1, characterized in that, In step S6, when registering and fusing multi-frame point clouds, a combination of iterative nearest point registration and registration based on normal vector features is used; when fitting the weld space trajectory, non-uniform rational B-spline curve fitting is used, and the theoretical attitude field of the welding gun is generated based on the tangent vector and normal vector of the fitted curve. In S7, the welding working state includes at least the normal working state, the state to be corrected state, and the alarm state.

7. The machine vision-based welding robot operation monitoring method as described in claim 1, characterized in that, In step S7, the working monitoring parameters include at least lateral deviation, height deviation, posture deviation, weld continuity, and visual interference index; When the lateral deviation, height deviation, and posture deviation are all within the preset first-level threshold range, and the weld continuity is not lower than the continuity threshold and the visual interference index is not higher than the interference threshold, it is judged as a normal working state. When at least one monitoring parameter exceeds the first-level threshold but does not exceed the second-level threshold, it is determined to be in a state requiring correction. An alarm is triggered when at least one monitoring parameter exceeds the secondary threshold or when no valid weld features are extracted for multiple consecutive frames.

8. The machine vision-based welding robot operation monitoring method as described in claim 6 or 7, characterized in that, It also includes: S8. When the judgment result is a state to be corrected, the work monitoring parameters are compensated and predicted based on the visual feedback delay and the robot motion response delay, and the robot end correction command is output; when the judgment result is an alarm state, the alarm information is output and the abnormal image, abnormal location and process parameters are recorded.

9. The machine vision-based welding robot operation monitoring method as described in claim 8, characterized in that, In step S8, the visual feedback delay includes image acquisition delay, image processing delay, and communication delay, and the robot motion response delay includes controller processing delay and actuator inertial response delay. Based on the visual feedback delay and robot motion response delay, a discrete state prediction model is established, and after making forward predictions on the current working monitoring parameters, a robot end effector correction command is output.

10. A machine vision-based welding robot work monitoring system, characterized in that, It includes: The task initialization module is used to read the theoretical welding trajectory, welding process parameters and monitoring thresholds of the workpiece to be welded, and to establish a set of monitoring objects for the welding task. The calibration modeling module is used to perform camera intrinsic parameter calibration, hand-eye calibration, and welding tool center point calibration to establish a unified coordinate transformation relationship; The vision acquisition module is used to acquire image sequences before and during welding, and simultaneously acquire the pose information of the industrial robot and welding process parameter information. The image preprocessing module is used to preprocess image sequences and construct dynamic regions of interest. The feature extraction module is used to extract at least one feature from the weld centerline, groove edge, structured light stripes, molten pool contour, and welding torch projection, and to calculate the working monitoring parameters. The 3D reconstruction module is used to construct a 3D point cloud of the welding area and generate a 3D reference trajectory of the weld and a theoretical attitude field of the welding torch. The status assessment module is used to integrate weld features, work monitoring parameters, weld three-dimensional reference trajectory, actual pose of industrial robot and welding process parameters, and determine the current welding work status. The compensation output module is used to output robot end effector correction commands when the current welding working state is in the state to be corrected. The alarm recording module is used to output alarm information and record abnormal images, abnormal locations, and process parameters when the current welding operation is in an alarm state.

Images