A welding molten pool state monitoring and control method and device based on visual feedback

Abstract

The application belongs to the technical field of automatic welding and industrial robot control, and relates to a welding molten pool state monitoring and control method and device based on visual feedback. The method comprises the following steps: controlling an industrial camera to perform position translation, so that a to-be-welded area enters the effective field of view range of the industrial camera; in a welding process, continuously collecting images of the welding area to obtain multiple welding images; performing weighted fusion on the multiple welding images to obtain a fused image, and performing preprocessing on the fused image to obtain a preprocessed image; performing molten pool state recognition based on the preprocessed image to obtain current molten pool state parameters and corresponding welding defect information; comparing the current molten pool state parameters with preset target parameters of the molten pool to obtain a state deviation, and generating corresponding driving instructions according to the state deviation; and performing multi-stage posture adjustment and fine positioning control on the industrial camera based on the driving instructions, so that the welding molten pool is kept at an optimal observation viewing angle.

Description

Technical Field

[0001] This application belongs to the field of automated welding and industrial robot control technology, and in particular relates to a method and device for monitoring and controlling the state of welding molten pool based on visual feedback. Background Technology

[0002] In automated and robotic welding processes, the stability of welding quality highly depends on the precise monitoring of the weld pool's state. The morphology, temperature, and dynamic fluctuations of the weld pool directly affect the weld formation quality and mechanical properties. However, existing welding systems still have significant shortcomings in practical applications. First, the single perspective is a prominent issue. Most weld pool monitoring devices only use fixed cameras for observation, and the weld pool is easily obscured by welding torches, spatter, or smoke, resulting in incomplete image information and affecting the accuracy of weld pool state recognition. Second, there is insufficient independence. Existing vision devices are usually directly integrated with the welding robot body, lacking cross-equipment versatility and difficulty adapting to different welding stations or different brands of equipment, thus limiting the realization of flexible production. Third, real-time performance and accuracy are limited. High-temperature welding environments are subject to interference such as strong reflections, smoke, and arc flashes, which easily affect traditional image processing algorithms. At the same time, single-view acquisition lacks redundant information, resulting in insufficient accuracy in weld pool feature extraction and difficulty in achieving high-precision real-time control. In addition, existing systems lack the ability to respond quickly to dynamic deviations during the welding process and cannot adjust the welding trajectory and process parameters in a timely manner according to the weld pool state, reducing the stability and consistency of welding quality.

[0003] Therefore, there is an urgent need for a monitoring and control device that can be deployed independently, supports controllable automatic adjustment of the viewing angle, and has high-precision real-time molten pool status recognition and feedback functions, so as to achieve accurate dynamic monitoring of the welding molten pool and timely adjust the welding trajectory and process parameters, thereby improving welding quality and production efficiency.

[0004] In summary, how to achieve high-precision, real-time, and adjustable-viewpoint monitoring and control of the molten pool condition in complex welding environments has become an urgent technical problem to be solved. Summary of the Invention

[0005] To address the problems of difficulty in real-time and accurate monitoring of the molten pool state, limited viewing angle, and difficulty in timely detection of welding defects in existing welding processes, this application proposes a method and device for monitoring and controlling the molten pool state based on visual feedback. By continuously acquiring images of the welding area, fusing and preprocessing fused images, and real-time monitoring of molten pool state parameters and defect information identification, as well as multi-level industrial camera attitude adjustment and precise positioning control based on the deviation between the molten pool state and the preset target state, the method achieves continuous optimization of the molten pool viewing angle and closed-loop dynamic control of the welding process. This enables high-precision monitoring of the molten pool state, timely identification of defects, and intelligent adaptive adjustment of welding quality throughout the entire welding process.

[0006] To achieve the above objectives, the first aspect of this application provides a method for monitoring and controlling the state of a weld pool with controllable adjustment of the viewing angle, comprising the following steps:

[0007] Step S1: Control the industrial camera to move its position so that the area to be welded enters the effective field of view of the industrial camera.

[0008] Step S2: During the welding process, control the industrial camera to continuously acquire images of the welding area and obtain multiple welding images;

[0009] Step S3: Weighted fusion of multiple welding images is performed to obtain a fused image, and the fused image is preprocessed to obtain a preprocessed image;

[0010] Step S4: Based on the preprocessed image, identify the state of the molten pool to obtain the current molten pool state parameters and the corresponding welding defect information;

[0011] Step S5: Compare the current molten pool state parameters with the preset target parameters of the molten pool to obtain the state deviation, and generate the corresponding drive command based on the state deviation.

[0012] Step S6: Perform attitude and position control on the industrial camera based on the drive command to keep the weld pool at the best viewing angle;

[0013] Step S7: Repeat steps S2 to S6 to form a closed-loop tracking of the weld pool based on visual feedback.

[0014] In some embodiments, the method for controlling the industrial camera to perform position translation is as follows:

[0015] Based on the spatial relationship between the spatial coordinates of the area to be welded and the current position information of the industrial camera, the target position that the industrial camera needs to reach in order to cover the area to be welded is calculated.

[0016] Based on the field of view parameters of the industrial camera, determine the effective imaging area of ​​the industrial camera and obtain the geometric center of the effective imaging area;

[0017] The center coordinates of the area to be welded are compared with the geometric center of the effective imaging area to obtain the translation compensation amount used to correct the position of the industrial camera.

[0018] The translation compensation is decomposed into displacement components along the orthogonal coordinate axes, and the industrial camera is controlled to perform position translation based on the displacement components, so that the image of the area to be welded falls within the effective imaging area of ​​the industrial camera.

[0019] In some embodiments, the welding pool condition monitoring and control method further includes an industrial camera hand-eye calibration step, specifically:

[0020] Before the welding operation begins, the industrial camera is calibrated using a calibration board to obtain the camera's focal length, principal point coordinates, and distortion coefficient.

[0021] Control the industrial camera to acquire images of the calibration board in multiple different poses;

[0022] Based on the image from the calibration plate, the rotation matrix and translation vector between the industrial camera coordinate system and the mechanical support mechanism base coordinate system are obtained, and the hand-eye calibration matrix is ​​established.

[0023] The depth information between the industrial camera and the calibration board is obtained, and the depth information is combined with the two-dimensional coordinates of the industrial camera imaging to obtain the final hand-eye calibration result.

[0024] In some embodiments, the fused image is preprocessed to obtain a preprocessed image, including the following steps:

[0025] Denoising and filtering are performed on the fused image to obtain the denoised image;

[0026] Morphological processing is performed on the denoised image to obtain the image after removing splash interference;

[0027] Image enhancement processing is performed on the image after removing splash interference to obtain the enhanced image;

[0028] The enhanced image is then subjected to region fusion and grayscale continuity correction to obtain a preprocessed image.

[0029] In some embodiments, molten pool state identification is performed based on the preprocessed image, and the current molten pool state parameters are extracted, including the following steps:

[0030] The preprocessed image is subjected to detection or segmentation of the molten pool region, and the molten pool contour information is extracted based on the detection or segmentation results to obtain the corresponding molten pool region mask;

[0031] The geometric centroid position of the molten pool region is calculated based on the molten pool region mask to obtain the current molten pool center position. The molten pool center position is then compared with the preset weld center line to obtain the weld offset of the molten pool relative to the weld center line.

[0032] Temporal analysis was performed on multiple preprocessed images acquired in succession to obtain the surface motion information of the molten pool;

[0033] The current molten pool state parameters are obtained by summarizing the information on the center position of the molten pool, the outline of the molten pool, the offset of the weld, and the surface movement of the molten pool.

[0034] In some embodiments, the method for identifying corresponding welding defect information is as follows:

[0035] Based on the preprocessed image, the grayscale distribution characteristics, geometric morphology characteristics and temporal variation characteristics of the molten pool area are analyzed, and feature quantities for welding defect judgment are extracted.

[0036] When the feature quantity meets the judgment condition for any welding defect, the corresponding welding defect type is determined and the welding defect identification result is generated.

[0037] In some embodiments, step S5 includes the following steps:

[0038] The desired state vector is constructed based on the preset target state parameters, and the actual state vector is constructed based on the current molten pool state parameters.

[0039] Based on the actual state vector and the target state vector, the molten pool state deviation is calculated, including the molten pool center position deviation, the molten pool contour feature deviation, and the weld offset deviation.

[0040] The molten pool state deviation is converted into a camera velocity vector using the interaction matrix, and then adjusted according to the adaptive gain.

[0041] The camera velocity vector is mapped to the velocity or position setpoint of each execution unit to form drive commands.

[0042] In some embodiments, the method for performing attitude and position control on an industrial camera based on drive commands is as follows:

[0043] Based on drive commands, the industrial camera is controlled to move rapidly across the plane, so that the weld pool enters the effective field of view of the industrial camera.

[0044] Based on the drive commands, the rotation and pitch angle of the industrial camera are adjusted so that the observation direction of the industrial camera is aligned with the molten pool area.

[0045] Based on drive commands, the industrial camera is precisely positioned and the lens is focused, enabling the industrial camera to achieve high-precision attitude control.

[0046] Real-time depth information between the industrial camera and the weld pool is acquired, and a three-dimensional surface model of the weld pool is constructed based on the depth information.

[0047] Based on the three-dimensional surface model, the spatial pose deviation of the industrial camera relative to the weld pool is estimated, and the corresponding compensation control quantity is calculated to correct the attitude and position of the industrial camera.

[0048] The focus position of the industrial camera lens is adjusted in real time based on depth information to ensure that the image is always clear.

[0049] In some embodiments, the welding pool condition monitoring and control method further includes the following steps:

[0050] Step S8: During the closed-loop tracking process, when the image quality is lower than the set image quality threshold or an arc overexposure occurs, the imaging parameters of the industrial camera are automatically adjusted.

[0051] Step S9: The molten pool status parameters and welding defect information are transmitted to the welding robot controller in real time, and the welding trajectory or process parameters are dynamically adjusted accordingly until the welding process is completed.

[0052] A second aspect of this application provides a welding pool condition monitoring and control device with controllable viewing angle, used to implement the above-mentioned welding pool condition monitoring and control method, comprising:

[0053] A field-view controllable vision acquisition module includes at least one industrial camera for continuous image acquisition of the welding area and to obtain multiple welding images.

[0054] A multi-degree-of-freedom mechanical support mechanism is used to support the industrial camera and adjust and control the position and attitude of the industrial camera, so that the welding area is always kept within the effective field of view of the industrial camera and at the best observation angle.

[0055] The image processing and molten pool state recognition module is used to preprocess multiple welding images, extract molten pool state parameters, and identify corresponding welding defect information.

[0056] The vision servo control module generates control commands based on the deviation between the molten pool state parameters and the target state parameters, and performs attitude and position control on the industrial camera based on the drive commands, so that the welding molten pool is always kept at the best viewing angle.

[0057] Compared with the prior art, the advantages and positive effects of this application are as follows: by continuously acquiring images of the welding area through an industrial camera, image fusion and preprocessing, molten pool status identification and defect judgment, combined with multi-level attitude adjustment and precise positioning control, the welding molten pool is always kept at the best viewing angle, thereby realizing real-time status monitoring and dynamic control of the entire welding process. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of a visually adjustable welding pool condition monitoring and control device disclosed in an embodiment of this application.

[0059] Figure 2 This is a schematic diagram of a visually adjustable welding pool condition monitoring and control device disclosed in an embodiment of this application.

[0060] Figure 3 This is a schematic flowchart of a welding pool condition monitoring and control method with controllable viewing angle disclosed in an embodiment of this application.

[0061] In the picture:

[0062] 1-X-axis translation drive motor; 2-Y-axis translation slide rail; 3-X-axis translation slide rail; 4-Z-axis rotation platform; 5-End swing joint; 6-Industrial camera; 7-Micro-adjustment platform; 8-End swing drive motor; 9-Z-axis rotation drive motor; 10-Y-axis translation drive motor; 11-Laser displacement sensor. Detailed Implementation

[0063] The present application will now be described in detail through exemplary embodiments. However, it should be understood that, without further description, elements, structures, and features in one embodiment may be advantageously incorporated into other embodiments.

[0064] like Figure 1 and Figure 2 As shown, a visually adjustable welding pool condition monitoring and control device includes:

[0065] The view-controllable vision acquisition module includes an industrial camera 6, which is used to continuously acquire images of the welding area and obtain multiple welding images.

[0066] A multi-degree-of-freedom mechanical support mechanism is used to support an industrial camera and adjust and control its position and orientation, ensuring that the welding area is always within the effective field of view of the industrial camera and at the optimal observation angle. It includes: a cross-shaped slide rail, a column mounted on the cross-shaped slide rail, a Z-axis rotating platform 4 mounted on the column, and an end-swing joint 5 connecting the Z-axis rotating platform 4. The end-swing joint 5 has a micro-adjustment platform 7 at its end, and the industrial camera 6 is fixed to the micro-adjustment platform 7. The cross-shaped slide rail includes an X-axis translation drive motor 1, an X-axis translation slide rail 3, a Y-axis translation drive motor 10, and a Y-axis translation slide rail 2. The Y-axis translation drive motor 10 drives the Y-axis translation slide rail 2; the X-axis translation drive motor 1 drives the X-axis translation slide rail 3; a Z-axis rotation drive motor 9 is mounted on the column for driving rotation; and an end-swing drive motor 8 is mounted on the Z-axis rotating platform 4 for driving the end-swing joint 5 to swing. A laser displacement sensor 11 (or TOF sensor) is mounted on one side of the industrial camera 6 to obtain depth information;

[0067] The image processing and molten pool state recognition module is used to receive images acquired by the industrial camera and perform image preprocessing, and extract molten pool features based on the preprocessed images and generate molten pool state parameters, including but not limited to molten pool center position, molten pool contour, weld offset and molten pool surface motion information.

[0068] The visual servo control module is used to compare the molten pool state parameters with preset target parameters to obtain the error, and calculate the corresponding drive command according to the error to control the multi-degree-of-freedom mechanical support mechanism to automatically adjust the viewing angle of the single industrial camera, so as to realize closed-loop tracking of molten pool vision.

[0069] The communication interface module is used to transmit the molten pool status parameters or control commands to the welding robot controller, welding machine control unit or host computer in real time via industrial bus or network, and to receive synchronization or coordination signals from the welding robot controller.

[0070] The aforementioned vision-adjustable welding pool condition monitoring and control device achieves continuous, high-precision acquisition and real-time tracking of welding pool images through the collaborative operation of an industrial camera and a multi-degree-of-freedom mechanical support mechanism. The industrial camera achieves translation, rotation, and micro-adjustment through a cross-shaped slide rail, a Z-axis rotating platform, and an end effector swing joint. Combined with a laser displacement sensor or a TOF sensor to obtain depth information, it achieves multi-degree-of-freedom attitude control. The image processing and welding pool condition recognition module preprocesses the acquired images and extracts the center position of the welding pool, the outline of the welding pool, the weld offset, and the surface motion information of the welding pool to generate welding pool condition parameters. The vision servo control module automatically adjusts the industrial camera's viewing angle based on the error calculation drive command between the welding pool condition parameters and preset target parameters, achieving closed-loop tracking of the welding pool vision. The communication interface module is used to transmit the welding pool condition parameters and control commands to the welding robot controller or host computer in real time and receive synchronization or coordination signals. On this basis, high-precision monitoring, dynamic closed-loop control, and visual management of the welding pool condition are achieved, thereby improving welding quality, ensuring process stability, and meeting the long-term operation requirements in complex welding environments.

[0071] Specifically, the welding pool condition monitoring and control device supports a "coarse-fine" viewing angle adjustment strategy: first, coarse positioning is performed by the X-axis translation slide rail 3 and the Y-axis translation slide rail 2 to bring the weld pool into the center of the field of view of the industrial camera 6; then, the Z-axis rotation platform 4 and the end swing joint 5 are used for angle approximation; finally, fine positioning and focusing are performed by the micro-adjustment platform 7. The velocity vector of the industrial camera 6... velocity vector of multi-degree-of-freedom mechanical support mechanism Through Jacobian matrix Mapping: ;in, For the 6-velocity vector of the industrial camera; The velocity vector of the multi-degree-of-freedom mechanical support mechanism; The translation speed of industrial camera 6 in three-dimensional space; The rotational speed of industrial camera 6 in three-dimensional space; The speed of the X-axis translation slide rail 3; The speed of the Y-axis translation slide rail 2; The angular velocity of the Z-axis rotating platform 4; The angular velocity of the end-point swing joint 5; For fine-tuning the displacement of the micro-adjustment platform 7 in the X direction; For fine-tuning the displacement of the micro-adjustment platform 7 in the Y direction; Jacobian matrix Describe the sensitivity of the degree-of-freedom changes of the multi-degree-of-freedom mechanical support mechanism to the spatial position and attitude changes of the industrial camera 6.

[0072] The welding pool condition monitoring and control device described in this application achieves continuous and precise capture and tracking of the welding pool through the collaborative operation of a multi-degree-of-freedom mechanical support mechanism and an industrial camera. The support mechanism adopts a "coarse-fine" perspective adjustment strategy. First, coarse positioning is performed by the X-axis and Y-axis translational slide rails to bring the welding pool into the center of the industrial camera's field of view. Then, the Z-axis rotation platform and the end-effector swing joint approximate the angle. Finally, the micro-adjustment platform completes the fine positioning and lens focusing. The three-dimensional translational and rotational speeds of the industrial camera are mapped to the multi-degree-of-freedom velocity vectors of the mechanical support mechanism through a Jacobian matrix, achieving high-precision control of the industrial camera's spatial position and attitude. The Jacobian matrix describes the sensitivity of changes in mechanical degrees of freedom to changes in the camera's spatial position and attitude. Based on this, precise perspective control, closed-loop tracking, and stable image acquisition of the welding pool are achieved, thereby improving the accuracy of welding observation, optimizing the welding pool condition monitoring effect, and meeting the long-term operation requirements in complex welding environments.

[0073] Specifically, the image processing and molten pool state recognition module includes an automatic exposure and HDR synthesis submodule ( ), Noise Reduction and Filtering Submodule (( And the image enhancement submodule (using histogram equalization or CLAHE: To adapt to the strong arc light and reflection interference during the welding process, the molten pool state recognition module employs a combination of a deep learning-based detection / segmentation network (e.g., a modified YOLO structure or UNet) and an optical flow analysis module (based on a constant brightness constraint equation) to achieve molten pool contour segmentation, centerline extraction, and surface velocity estimation, and to calculate the camera image viewpoint position of the molten pool. The molten pool state recognition module further includes a fault / defect detection submodule for real-time identification of defects such as porosity, incomplete penetration, slag inclusions, and excessive spatter, triggering alarms or recording time-series images, wherein: This represents a fused image obtained by weighted fusion of different exposure images, used to adapt to strong welding arc light and reflection interference; For industrial cameras 6 in the first Original welding images acquired under specific exposure conditions; For industrial camera 6, the exposure weighting coefficient is used. These are the pixels of the welding image after noise reduction processing; These are the pixel values ​​within the noise reduction window, used for weighted calculations. This represents the contribution of each neighboring pixel to the center pixel; For the filter window index; To output enhanced welding images; The result of the image enhancement submodule processing; To enhance image pixels.

[0074] The welding pool condition monitoring and control device described in this application achieves high-precision visual acquisition and real-time analysis of the welding area through an image processing and welding pool condition recognition module. The module includes an automatic exposure and HDR synthesis submodule, which weights and fuses images from different exposures to generate a fused image to adapt to strong arc light and reflection interference during welding; a noise reduction and filtering submodule weights image pixels to remove noise interference; an image enhancement submodule uses histogram equalization or CLAHE to improve image contrast; the welding pool condition recognition uses a deep learning detection / segmentation network combined with optical flow analysis to achieve welding pool contour segmentation, centerline extraction, and surface velocity estimation, and calculates the position of the welding pool in the camera's view; a fault / defect detection submodule is used to identify defects such as porosity, incomplete penetration, slag inclusion, and excessive spatter in real time, and trigger alarms or record time-series images; based on this, it achieves automated, real-time, and visual management of continuous acquisition of welding pool images, extraction of condition parameters, and defect detection, thereby improving the monitoring accuracy of the welding process, ensuring welding quality, and meeting the requirements for long-term stable operation in complex welding environments.

[0075] Specifically, the device also includes a laser ranging module or a TOF sensor to provide depth information between the camera and the weld pool, which is used for hand-eye calibration correction and 3D reconstruction of PBVS.

[0076] Specifically, the visual servo control module employs image-based visual servoing (IBVS) and / or attitude-based visual servoing (PBVS), and uses an interaction matrix. L s Or the hand-eye calibration matrix will account for the error in the image domain ( ) is converted into drive commands for the multi-degree-of-freedom mechanical support mechanism, wherein, Current molten pool state parameters With the target molten pool state parameters The difference.

[0077] Specifically, the visual servo control module includes an adaptive control strategy: when the image quality is detected to be lower than a set threshold or when arc overexposure occurs, the camera exposure, shutter speed and gain are automatically adjusted, and if necessary, the system switches to infrared imaging or reduces the servo gain to ensure system stability.

[0078] Specifically, the control law of the visual servo control module includes: based on the interaction matrix L s The linear control law, ,in, For adaptive gain, The pseudo-inverse of the interaction matrix is ​​obtained by using the hand-eye calibration matrix and the device Jacobian matrix. The mapping is applied to the speed / position settings of each execution unit.

[0079] The aforementioned visual servo control module of this application realizes closed-loop tracking of the molten pool and automatic viewing angle adjustment; it adopts image-based or posture-based visual servoing, converting the molten pool state error into mechanical support mechanism drive commands through an interaction matrix or hand-eye calibration matrix, and combines an adaptive control strategy to automatically adjust camera parameters or servo gain when image quality deteriorates or arc light is overexposed; based on a linear control law, it maps the camera velocity vector to the velocity and position setpoints of each execution unit, realizing high-precision, stable control and real-time monitoring of the molten pool viewing angle during welding.

[0080] Specifically, the communication interface module supports at least one industrial protocol: EtherCAT, CAN, RS485 or Industrial Ethernet, and supports data uploading and historical recording with MES or host computer.

[0081] like Figure 3 As shown, a method for monitoring and controlling the state of a weld pool with controllable viewing angle, based on the aforementioned weld pool state monitoring and control device, includes the following steps:

[0082] Step S1: Control the industrial camera to perform a position translation so that the area to be welded enters the effective field of view of the industrial camera.

[0083] Step S2: During the welding process, control the industrial camera to continuously acquire images of the welding area and obtain multiple welding images;

[0084] Step S3: Weighted fusion of multiple welding images to obtain a fused image, and preprocessing of the fused image to obtain a preprocessed image;

[0085] Step S4: Based on the preprocessed image, identify the state of the molten pool to obtain the current molten pool state parameters and the corresponding welding defect information;

[0086] Step S5: Compare the current molten pool state parameters with the preset target parameters of the molten pool to obtain the state deviation, and generate the corresponding drive command based on the state deviation.

[0087] Step S6: Perform multi-level attitude adjustment and precise positioning control on the industrial camera based on the drive command to keep the weld pool at the best viewing angle.

[0088] Step S7: Repeat steps S2 to S6 to form a closed-loop tracking of the weld pool based on visual feedback.

[0089] Step S8: During the closed-loop tracking process, when the image quality is lower than the set image quality threshold or an arc overexposure occurs, the imaging parameters of the industrial camera are automatically adjusted.

[0090] Step S9: The molten pool status parameters and welding defect information are transmitted to the welding robot controller in real time, and the welding trajectory or process parameters are dynamically adjusted accordingly until the welding process is completed.

[0091] The welding pool condition monitoring and control method described in this application achieves continuous image acquisition, condition parameter identification, and defect detection in the welding area through the collaborative operation of an industrial camera and a multi-degree-of-freedom mechanical support mechanism. The method includes translating the industrial camera to the effective field of view, continuous image acquisition, image fusion and preprocessing, weld pool condition identification and defect judgment, condition deviation calculation and drive command generation, and multi-level attitude adjustment and precise positioning control based on the drive commands to achieve closed-loop tracking of the weld pool from the optimal observation angle. During the closed-loop process, imaging parameters are automatically adjusted to cope with image quality degradation or arc overexposure, and the condition parameters and defect information are transmitted to the welding robot controller in real time to dynamically optimize the welding trajectory and process parameters, thereby achieving automated, continuous, and visualized management of the entire welding process and improving welding quality and system stability.

[0092] In some embodiments, the method for controlling the industrial camera to perform position translation is as follows:

[0093] Based on the spatial relationship between the spatial coordinates of the area to be welded and the current position information of the industrial camera, the target position that the industrial camera needs to reach in order to cover the area to be welded is calculated.

[0094] Based on the field of view parameters of the industrial camera, determine the effective imaging area of ​​the industrial camera and obtain the geometric center of the effective imaging area;

[0095] The center coordinates of the area to be welded are compared with the geometric center of the effective imaging area to obtain the translation compensation amount used to correct the position of the industrial camera.

[0096] The translation compensation is decomposed into displacement components along the orthogonal coordinate axes, and the industrial camera is controlled to perform position translation based on the displacement components, so that the image of the area to be welded in the industrial camera falls into the effective imaging area.

[0097] The method for controlling the position translation of an industrial camera in this application determines the required camera position to cover the target by calculating the spatial relationship between the spatial coordinates of the area to be welded and the current position of the camera; the effective imaging area and its geometric center are obtained according to the camera's field of view parameters; the translation compensation amount is obtained by comparing the center of the area to be welded with the geometric center and decomposed into displacement components along the orthogonal coordinate axes; the industrial camera is driven to perform precise translation, so that the area to be welded is always located within the effective imaging area in the camera, thereby ensuring the accuracy of image acquisition and the reliability of molten pool status monitoring during the welding process.

[0098] In some embodiments, the welding pool condition monitoring and control method further includes an industrial camera hand-eye calibration step, specifically:

[0099] Before the welding operation begins, the industrial camera is calibrated using a calibration board to obtain the camera's focal length, principal point coordinates, and distortion coefficient.

[0100] Control the industrial camera to acquire images of the calibration board in multiple different poses, where there is sufficient rotational component difference between adjacent poses;

[0101] Based on the image from the calibration plate, the rotation matrix and translation vector between the industrial camera coordinate system and the mechanical support mechanism base coordinate system are obtained, and the hand-eye calibration matrix is ​​established.

[0102] The depth information between the industrial camera and the calibration plate is obtained, and the depth information is combined with the two-dimensional coordinates of the industrial camera imaging to obtain the final hand-eye calibration result.

[0103] The welding pool condition monitoring and control method described in this application achieves precise correlation between the camera coordinate system and the mechanical support mechanism base coordinate system through an industrial camera hand-eye calibration step. Before welding, a calibration plate is used to calibrate the camera's intrinsic parameters to obtain the focal length, principal point coordinates, and distortion coefficients. Images of the calibration plate are acquired in multiple different poses to ensure sufficient differences in rotational components. Based on the calibration images, a rotation matrix and translation vector are calculated to establish a hand-eye calibration matrix. The final hand-eye calibration result is obtained by combining the depth information between the camera and the calibration plate with the two-dimensional imaging coordinates, thereby ensuring high precision and stability of industrial camera positioning and attitude control during the welding process.

[0104] In some embodiments, the fused image is preprocessed to obtain a preprocessed image, including the following steps:

[0105] Denoising and filtering are performed on the fused image to suppress image noise caused by environmental interference factors, resulting in a denoised image.

[0106] Morphological processing is performed on the denoised image to remove isolated bright spots caused by welding spatter, resulting in an image free of spatter interference.

[0107] Image enhancement processing is performed on the image after removing spatter interference. By adjusting the grayscale distribution of the image, the contrast and detail features of the molten pool area are enhanced to obtain the enhanced image.

[0108] The enhanced image is processed by region fusion and grayscale continuity correction. Bilinear interpolation is used to smooth the grayscale values ​​at the boundaries of each sub-region, eliminating grayscale discontinuities, and thus obtaining the preprocessed image.

[0109] The method for obtaining the preprocessed image described in this application preprocesses the fused image to achieve denoising, filtering, and morphological processing of the molten pool image, suppressing environmental interference and isolated spatter spots; then, image enhancement is used to improve the contrast and detail features of the molten pool area, and region fusion and grayscale continuity correction are used to smoothly transition the grayscale of the sub-region boundaries, thereby obtaining a clear, continuous preprocessed image that can be used for molten pool state recognition, realizing high-precision visual analysis and monitoring during the welding process.

[0110] In some embodiments, molten pool state identification is performed based on the preprocessed image, and the current molten pool state parameters are extracted, including the following steps:

[0111] The preprocessed image is subjected to detection or segmentation of the molten pool region, and the molten pool contour information is extracted based on the detection or segmentation results to obtain the corresponding molten pool region mask;

[0112] The geometric centroid position of the molten pool region is calculated based on the molten pool region mask to obtain the current molten pool center position. The molten pool center position is then compared with the preset weld center line to obtain the weld offset of the molten pool relative to the weld center line.

[0113] Temporal analysis is performed on multiple preprocessed images acquired in succession, and motion information of pixels in the molten pool region is extracted based on optical flow analysis method to obtain molten pool surface motion information;

[0114] The current molten pool state parameters are formed by summarizing the information on the center position of the molten pool, the outline of the molten pool, the offset of the weld, and the surface movement of the molten pool.

[0115] The method described above for extracting current molten pool state parameters in this application involves detecting and segmenting the molten pool region in a preprocessed image, extracting the molten pool contour and region mask, and calculating the offset between the geometric centroid of the molten pool and the weld centerline. Combined with temporal analysis of multiple frames of images, the surface motion information of the molten pool is obtained based on the optical flow method. Finally, the center position, contour, weld offset, and surface motion of the molten pool are summarized to form complete current molten pool state parameters, thereby achieving accurate visual monitoring and state recognition of the welding process.

[0116] In some embodiments, the method for identifying corresponding welding defect information is as follows:

[0117] Based on the preprocessed image, the grayscale distribution characteristics, geometric morphology characteristics and temporal variation characteristics of the molten pool area are analyzed, and feature quantities for welding defect judgment are extracted.

[0118] Based on the aforementioned characteristic quantities, porosity defects, incomplete penetration defects, slag inclusion defects, and excessive spatter defects are determined.

[0119] When the feature quantity satisfies the judgment condition of any welding defect, the corresponding welding defect type is determined and a welding defect identification result is generated;

[0120] The welding defect identification result is output and an alarm is triggered. At the same time, the time-series image data corresponding to the welding defect identification result is stored.

[0121] The method described above for identifying corresponding welding defect information in this application extracts feature quantities for judging welding defects such as porosity, incomplete penetration, slag inclusion, and excessive spatter by analyzing the grayscale, geometric shape, and temporal change characteristics of the molten pool area in the preprocessed image. When the feature quantity meets any defect judgment condition, the corresponding welding defect identification result is generated and an alarm is triggered. At the same time, the relevant temporal images are stored to realize real-time defect monitoring and recording of the welding process.

[0122] In some embodiments, step S5 includes the following steps:

[0123] The desired state vector is constructed based on the preset target state parameters, and the actual state vector is constructed based on the current molten pool state parameters.

[0124] Based on the actual state vector and the target state vector, the molten pool state deviation is calculated, including the molten pool center position deviation, the molten pool contour feature deviation, and the weld offset deviation.

[0125] The molten pool state deviation is converted into a camera velocity vector using the interaction matrix, and then adjusted according to the adaptive gain.

[0126] The camera velocity vector is mapped to the velocity or position setpoint of each execution unit to form drive commands.

[0127] In step S5 of this application, the center position, contour features and weld offset deviation are calculated by constructing a vector from the preset target state and the current molten pool state. The deviation is then converted into a camera velocity vector using an interaction matrix. After adaptive gain adjustment, the vector is mapped to each execution unit to form a drive command, thereby realizing precise attitude control of the industrial camera and closed-loop adjustment of the molten pool state.

[0128] In some embodiments, the method for performing multi-level attitude adjustment and fine positioning control of an industrial camera based on driving commands is as follows:

[0129] Based on drive commands, the industrial camera is controlled to move rapidly across the plane, so that the weld pool enters the effective field of view of the industrial camera.

[0130] Based on the drive commands, the rotation and pitch angle of the industrial camera are adjusted so that the observation direction of the industrial camera is aligned with the molten pool area.

[0131] Based on drive commands, the industrial camera is precisely positioned and the lens is focused, enabling the industrial camera to achieve high-precision attitude control.

[0132] Real-time depth information between the industrial camera and the weld pool is acquired, and a three-dimensional surface model of the weld pool is constructed based on the depth information.

[0133] Based on the three-dimensional surface model, the spatial pose deviation of the industrial camera relative to the weld pool is estimated, and the corresponding compensation control quantity is calculated to correct the attitude and position of the industrial camera.

[0134] The focus position of the industrial camera lens is adjusted in real time based on the depth information to ensure that the image is always clear.

[0135] The method described in this application for performing multi-level attitude adjustment and precise positioning control of an industrial camera based on drive commands achieves high-precision attitude control of the industrial camera over the molten pool by performing multi-level translation, rotation, and pitch adjustments of the industrial camera based on drive commands, combined with precise position adjustment and lens focusing. At the same time, it acquires depth information in real time to construct a three-dimensional surface model of the molten pool, and compensates and corrects for spatial pose deviations to ensure that the industrial camera always maintains a clear field of view and achieves continuous and accurate observation of the molten pool.

[0136] In some embodiments, the method for detecting image quality is as follows:

[0137] The average gradient magnitude or the variance of the Laplacian response of the preprocessed image is calculated as an image sharpness evaluation index. When the image sharpness evaluation index is lower than the preset sharpness threshold, the image is judged to be blurry.

[0138] The proportion of pixels with gray values ​​not less than the gray saturation value in the preprocessed image is counted. When the proportion exceeds the preset overexposure threshold, the image is determined to be overexposed.

[0139] A large area of ​​grayscale saturation caused by direct arc light is detected in the molten pool area and its vicinity. When the saturation area seriously affects the extraction of molten pool state parameters, it is determined that arc light overexposure has occurred.

[0140] Calculate the overall average brightness value of the preprocessed image. When the average brightness value makes the details in the melt pool area indistinguishable, the image is determined to be underexposed.

[0141] The method for detecting image quality described in this application determines the image sharpness, overexposure, arc overexposure, and underexposure by calculating the average gradient amplitude or Laplacian variance, grayscale saturation ratio, and overall average brightness of the preprocessed image. This enables real-time evaluation and dynamic adjustment of the molten pool image quality, ensuring the accuracy of molten pool state parameter extraction and the reliability of welding monitoring.

[0142] In some embodiments, the specific adjustment strategy for automatically adjusting the imaging parameters of the industrial camera is as follows: when overexposure or arc overexposure is detected, the exposure time is reduced and the gain is decreased according to a preset adjustment step size. If the overexposure cannot be eliminated after adjustment, the lens aperture is further reduced. When underexposure is detected, the exposure time is increased or the gain is increased according to a preset adjustment step size. When the above conventional adjustment still cannot obtain satisfactory image quality, the infrared imaging mode is switched to avoid strong arc interference in the visible light band.

[0143] The specific adjustment strategy for automatically adjusting the imaging parameters of the industrial camera described in this application automatically adjusts the imaging parameters of the industrial camera based on the image quality detection results, including adjusting the exposure time, gain and lens aperture, and switching to infrared imaging mode when necessary, so as to maintain image clarity under overexposure, underexposure or strong arc light interference, and ensure the continuity and accuracy of molten pool condition monitoring.

[0144] In some embodiments, dynamic adjustment of the welding trajectory includes:

[0145] Lateral trajectory correction: When the deviation of the center position of the molten pool exceeds the preset threshold, the welding torch is controlled to adjust its lateral position in the opposite direction of the offset, and the correction amount is proportional to the weld offset amount, so as to achieve weld tracking correction.

[0146] Longitudinal speed adjustment: When the deviation of the molten pool profile feature is greater than the preset target length, increase the welding travel speed; when the deviation of the molten pool profile feature is less than the preset target length, decrease the welding travel speed.

[0147] Trajectory attitude adjustment: Determine whether the welding torch attitude needs to be adjusted based on the surface motion information of the molten pool to ensure that the molten pool is in a stable flow state.

[0148] The dynamic adjustment method for welding trajectory described in this application achieves dynamic adjustment of the welding trajectory by analyzing the center position, contour features and surface motion information of the molten pool in real time. This includes lateral deviation correction to achieve weld tracking, longitudinal speed adjustment to maintain welding uniformity, and welding torch posture adjustment to ensure stable flow of the molten pool, thereby improving welding accuracy and process stability.

[0149] In some embodiments, dynamic adjustment of process parameters includes:

[0150] Welding current adjustment: Based on the deviation between the current molten pool outline area and the preset target area, the adjustment amount of the welding current is determined. When the molten pool outline area is smaller than the preset target area, the welding current is increased; when the molten pool outline area is larger than the preset target area, the welding current is decreased.

[0151] Arc voltage adjustment: The arc voltage is adjusted accordingly based on the deviation between the current molten pool width and the preset target width, so as to control the width of the molten pool.

[0152] Shielding gas flow rate regulation: When a tendency for porosity defects is detected and identified in the molten pool area, the supply flow rate of the shielding gas is increased through the welding machine control unit to reduce the probability of porosity defects.

[0153] Wire feed speed adjustment: It works in conjunction with welding current adjustment, and adjusts the wire feed speed synchronously according to changes in welding current, thereby maintaining the stability of the arc combustion state.

[0154] The dynamic adjustment method for the above-mentioned process parameters in this application achieves dynamic adjustment of welding process parameters by real-time monitoring of the molten pool contour area, width, and defect tendency. This includes synchronous adjustment of welding current and wire feed speed to maintain arc stability, adjustment of arc voltage to control molten pool width, and adjustment of shielding gas flow rate to reduce the probability of porosity defects, thereby improving welding quality and process stability.

[0155] The above embodiments are used to explain this application, not to limit it. Any modifications and changes made to this application within the spirit and scope of the claims shall fall within the protection scope of this application.

Claims

1. A method for monitoring and controlling the condition of a weld pool with adjustable viewing angle, comprising the following steps: Step S1: Control the industrial camera to move its position so that the area to be welded enters the effective field of view of the industrial camera. Step S2: During the welding process, control the industrial camera to continuously acquire images of the welding area and obtain multiple welding images; Step S3: Weighted fusion of multiple welding images is performed to obtain a fused image, and the fused image is preprocessed to obtain a preprocessed image; The fused image is preprocessed to obtain a preprocessed image, including the following steps: Denoising and filtering are performed on the fused image to obtain the denoised image; Morphological processing is performed on the denoised image to obtain the image after removing splash interference; Image enhancement processing is performed on the image after removing splash interference to obtain the enhanced image; The enhanced image is then subjected to region fusion and grayscale continuity correction to obtain a preprocessed image; Step S4: Based on the preprocessed image, identify the state of the molten pool to obtain the current molten pool state parameters and the corresponding welding defect information; Step S5: Compare the current molten pool state parameters with the preset target parameters of the molten pool to obtain the state deviation, and generate the corresponding drive command based on the state deviation. Step S6: Perform attitude and position control on the industrial camera based on the drive command to keep the weld pool at the best viewing angle; Step S7: Repeat steps S2 to S6 to form a closed-loop tracking of the weld pool based on visual feedback. It also includes the hand-eye calibration steps for industrial cameras, specifically: Before the welding operation begins, the industrial camera is calibrated using a calibration board to obtain the camera's focal length, principal point coordinates, and distortion coefficient. Control the industrial camera to acquire images of the calibration board in multiple different poses; Based on the image from the calibration plate, the rotation matrix and translation vector between the industrial camera coordinate system and the mechanical support mechanism base coordinate system are obtained, and the hand-eye calibration matrix is ​​established. The depth information between the industrial camera and the calibration board is obtained, and the depth information is combined with the two-dimensional coordinates of the industrial camera imaging to obtain the final hand-eye calibration result.

2. The method for monitoring and controlling the condition of the weld pool according to claim 1, characterized in that, The method for controlling the position translation of an industrial camera is as follows: Based on the spatial relationship between the spatial coordinates of the area to be welded and the current position information of the industrial camera, the target position that the industrial camera needs to reach in order to cover the area to be welded is calculated. Based on the field of view parameters of the industrial camera, determine the effective imaging area of ​​the industrial camera and obtain the geometric center of the effective imaging area; The center coordinates of the area to be welded are compared with the geometric center of the effective imaging area to obtain the translation compensation amount used to correct the position of the industrial camera. The translation compensation is decomposed into displacement components along the orthogonal coordinate axes, and the industrial camera is controlled to perform position translation based on the displacement components, so that the image of the area to be welded falls within the effective imaging area of ​​the industrial camera.

3. The method for monitoring and controlling the condition of the weld pool according to claim 1, characterized in that, Based on the preprocessed image, the molten pool state is identified, and the current molten pool state parameters are extracted, including the following steps: The preprocessed image is subjected to detection or segmentation of the molten pool region, and the molten pool contour information is extracted based on the detection or segmentation results to obtain the corresponding molten pool region mask; The geometric centroid position of the molten pool region is calculated based on the molten pool region mask to obtain the current molten pool center position. The molten pool center position is then compared with the preset weld center line to obtain the weld offset of the molten pool relative to the weld center line. Temporal analysis was performed on multiple preprocessed images acquired in succession to obtain the surface motion information of the molten pool; The current molten pool state parameters are obtained by summarizing the information on the center position of the molten pool, the outline of the molten pool, the offset of the weld, and the surface movement of the molten pool.

4. The method for monitoring and controlling the condition of the weld pool according to claim 1, characterized in that, The method for identifying the corresponding welding defect information is as follows: Based on the preprocessed image, the grayscale distribution characteristics, geometric morphology characteristics and temporal variation characteristics of the molten pool area are analyzed, and feature quantities for welding defect judgment are extracted. When the feature quantity meets the judgment condition for any welding defect, the corresponding welding defect type is determined and the welding defect identification result is generated.

5. The method for monitoring and controlling the condition of the weld pool according to claim 1, characterized in that, Step S5 includes the following steps: The desired state vector is constructed based on the preset target state parameters, and the actual state vector is constructed based on the current molten pool state parameters. Based on the actual state vector and the target state vector, the molten pool state deviation is calculated, including the molten pool center position deviation, the molten pool contour feature deviation, and the weld offset deviation. The molten pool state deviation is converted into a camera velocity vector using the interaction matrix, and then adjusted according to the adaptive gain. The camera velocity vector is mapped to the velocity or position setpoint of each execution unit to form drive commands.

6. The method for monitoring and controlling the condition of the weld pool according to claim 5, characterized in that, The method for performing attitude and position control of an industrial camera based on drive commands is as follows: Based on drive commands, the industrial camera is controlled to move rapidly across the plane, so that the weld pool enters the effective field of view of the industrial camera. Based on the drive commands, the rotation and pitch angle of the industrial camera are adjusted so that the observation direction of the industrial camera is aligned with the molten pool area. Based on drive commands, the industrial camera is precisely positioned and the lens is focused, enabling the industrial camera to achieve high-precision attitude control. Real-time depth information between the industrial camera and the weld pool is acquired, and a three-dimensional surface model of the weld pool is constructed based on the depth information. Based on the three-dimensional surface model, the spatial pose deviation of the industrial camera relative to the weld pool is estimated, and the corresponding compensation control quantity is calculated to correct the attitude and position of the industrial camera. The focus position of the industrial camera lens is adjusted in real time based on depth information to ensure that the image is always clear.

7. The method for monitoring and controlling the condition of the weld pool according to claim 1, characterized in that, The method for monitoring and controlling the condition of the weld pool also includes the following steps: Step S8: During the closed-loop tracking process, when the image quality is lower than the set image quality threshold or an arc overexposure occurs, the imaging parameters of the industrial camera are automatically adjusted. Step S9: The molten pool status parameters and welding defect information are transmitted to the welding robot controller in real time, and the welding trajectory or process parameters are dynamically adjusted accordingly until the welding process is completed.

8. A welding pool condition monitoring and control device with controllable viewing angle, used to implement the welding pool condition monitoring and control method as described in any one of claims 1-7, characterized in that, include: A field-view controllable vision acquisition module includes at least one industrial camera for continuous image acquisition of the welding area and to obtain multiple welding images. A multi-degree-of-freedom mechanical support mechanism is used to support the industrial camera and adjust and control the position and attitude of the industrial camera, so that the welding area is always kept within the effective field of view of the industrial camera and at the best observation angle. The image processing and molten pool state recognition module is used to preprocess multiple welding images, extract molten pool state parameters, and identify corresponding welding defect information. The vision servo control module generates control commands based on the deviation between the molten pool state parameters and the target state parameters, and performs attitude and position control on the industrial camera based on the drive commands, so that the welding molten pool is always kept at the best viewing angle.

Images