Positioning weld robot system based on active and passive visual sensing

Abstract

The invention discloses a locating weld robot system based on active and passive visual sensing, and relates to the technical field of robot welding, the locating weld robot system comprises a weld tracking module, the weld tracking module is used for monitoring a weld and a welding state and adjusting the position of a welding gun; the welding seam tracking module comprises a visual sensing unit, a deviation detection unit and an automatic deviation correction unit, the visual sensing unit is connected with the deviation detection unit through Bluetooth, and the deviation detection unit is connected with the automatic deviation correction unit through Bluetooth. By designing the welding seam tracking module, the automatic welding seam positioning function is achieved, the problem of welding seam position deviation is solved, the welding quality and efficiency of the welding seam robot are improved, and active and passive vision fusion is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of robotic welding, and specifically to a seam-tracking robotic system based on active and passive vision sensing. Background Art

[0002] As an important part of the modern industrial automation field, seam-tracking robotic systems are widely used in fields such as automotive, shipbuilding, aerospace, and steel structures. Compared with traditional manual welding, seam-tracking robotic systems have advantages such as high precision and high efficiency, reducing labor costs and improving operation safety.

[0003] Traditional seam-tracking robotic systems usually perform welding according to a pre-programmed fixed trajectory. When affected by factors such as workpiece deformation, assembly error, or environmental interference, they cannot make adaptive adjustments, resulting in the problem of seam position deviation, which affects the welding quality and increases the rework rate and production cost.

[0004] Patent CN114952098B discloses an intelligent seam-tracking system and welding robot based on machine vision guidance. The above patent realizes the further confirmation of the seam position, improves the efficiency of seam positioning, and realizes the detection of the quality after welding.

[0005] The above patent determines the specific position of the seam, solves the problem of inaccurate seam tracking of existing robots for seam tracking, but there is still room for optimization in terms of seam position deviation. This application calculates the deviation amount between the welding torch and the seam to solve the problem of seam position deviation.

[0006] Therefore, this application proposes a seam-tracking robotic system based on active and passive vision sensing that realizes the automatic control of the deviation between the welding torch and the seam. Summary of the Invention

[0007] The purpose of the present invention is to provide a seam-tracking robotic system based on active and passive vision sensing to solve the technical problems proposed in the above background art.

[0008] To achieve the above purpose, the present invention provides the following technical solution: A seam-tracking robotic system based on active and passive vision sensing, including a seam-tracking module, where the seam-tracking module is used to monitor the seam and welding state and adjust the position of the welding torch; The seam-tracking module includes: a vision sensing unit, a deviation detection unit, and an automatic deviation correction unit. The vision sensing unit is connected to the deviation detection unit via Bluetooth, and the deviation detection unit is connected to the automatic deviation correction unit via Bluetooth; The visual sensing unit scans the weld image through a binocular camera, generates a low-resolution point cloud, tracks the rough positioning area, projects a laser stripe in the rough positioning area, acquires the video image during the welding process, and generates a weld point cloud; The deviation detection unit extracts features from the video image data, extracts the upper and lower edge features of the welding torch and the upper and lower groove features of the weld, and combines the weld point cloud data to calculate the deviation amount between the center line of the welding torch and the center line of the weld; The automatic deviation correction unit calculates the direction and distance that the welding torch needs to be adjusted according to the deviation amount, adjusts the position and posture of the welding torch, and controls the deviation between the welding torch and the weld within a preset range.

[0009] Preferably, the weld tracking module is connected to a video frame splitting module via Bluetooth. The video frame splitting module is used to split the video image during the welding process into pictures frame by frame for image processing.

[0010] The video frame splitting module includes: a frame splitting processing unit, a field of view correction unit, and an image enhancement unit. The frame splitting processing unit is connected to the field of view correction unit via Bluetooth, the field of view correction unit is connected to the image enhancement unit via Bluetooth, and the visual sensing unit is connected to the frame splitting processing unit via Bluetooth; The frame splitting processing unit uses a video frame splitting algorithm to perform frame splitting processing on the video image during the welding process, splits the video stream into individual image frames frame by frame, and stores the image frames in the database in sequence; The field of view correction unit uses an image processing algorithm to identify the contour in the image frame, selects the area of interest, calculates the tilt angle of the image, and uses a geometric transformation algorithm to rotate and correct the image to restore the horizontal and vertical states of the area of interest; The image enhancement unit performs histogram analysis on the image frame, obtains the gray value range of the area of interest, and uses an image enhancement algorithm to perform enhancement processing on the image frame.

[0011] Preferably, the weld tracking module is connected to a weld defect module via Bluetooth. The weld defect module is used for weld defect detection; The weld defect module includes: an area extraction unit, a defect segmentation unit, and a defect identification unit. The area extraction unit is connected to the defect segmentation unit via Bluetooth, and the defect segmentation unit is connected to the defect identification unit via Bluetooth; The area extraction unit calculates the curvature value of each data in the weld point cloud, preliminarily segments the weld area and other areas, performs point cloud slicing processing on other areas, defines the weld area, and combines to obtain the complete weld area point cloud; The defect segmentation unit divides the weld area point cloud into grids, compares the number of points in the grid with a preset threshold, segments the burn-through area and the planar area, and uses a region growing segmentation algorithm to process the planar area to segment the defect area; The defect recognition unit extracts the point cloud features of the defect area, constructs an identification and classification model for weld defects, and automatically identifies and classifies the defects, including weld tumors, depressions, welding slag, and pores.

[0012] Preferably, the weld tracking module is connected to a sensing fusion module via Bluetooth. The sensing fusion module is used to integrate data from different sensors and adjust the parameters and working modes of the sensors. The sensing fusion module includes: a multi-dimensional fusion unit, a blind area compensation unit, and a dynamic collaboration unit. The multi-dimensional fusion unit is connected to the blind area compensation unit via Bluetooth, and the multi-dimensional fusion unit is connected to the dynamic collaboration unit via Bluetooth. The multi-dimensional fusion unit collects real-time monitoring data from the sensor network, including a force sensor, an arc sensor, and a temperature sensor, combines the image data during the welding process, and fuses the multi-dimensional data using a data fusion algorithm. The blind area compensation unit performs pattern recognition based on the multi-dimensional data, identifies the blind areas of the sensors, determines the missing situation of the multi-dimensional data, and uses the multi-dimensional data to compensate for the blind area information of other sensors. The dynamic collaboration unit analyzes the multi-dimensional data in real time, identifies the changes in the position, shape, and welding environment of the weld, and dynamically adjusts the working parameters of the sensor network and the welding parameters according to the specific requirements of the welding task.

[0013] Preferably, the weld defect module is connected to a weld trimming module via Bluetooth. The weld trimming module is used to process weld defects. The weld trimming module includes: a trimming planning unit and a trajectory planning unit. The trimming planning unit is connected to the trajectory planning unit via Bluetooth, and the trimming planning unit is connected to the defect recognition unit via Bluetooth. The trimming planning unit collects the results of weld defect identification and classification, including defect type, defect location, and defect size information, combines the physical properties of the welding material and the welding process parameters, and formulates a weld trimming strategy, including the selection of trimming tools, the planning of machining swing angles, and the planning of feed speeds. The trajectory planning unit calculates and optimizes the collision-free machining path of the trimming tool according to the weld trimming strategy and the weld defect data using a path planning algorithm.

[0014] Preferably, the image enhancement unit is connected to a dynamic compensation module via Bluetooth. The dynamic compensation module is used to monitor the light intensity of the welding area in real time and adjust the parameters of the binocular camera. The dynamic compensation module includes a light monitoring unit, a camera adjustment unit, and a quality assessment unit. The light monitoring unit is connected to the camera adjustment unit via Bluetooth, the camera adjustment unit is connected to the quality assessment unit via Bluetooth, and the image enhancement unit is connected to the quality assessment unit via Bluetooth. The light monitoring unit uses the light intensity sensors mounted on the binocular cameras to monitor the light intensity in the welding area in real time, analyze the changing trend of the light intensity, and identify the fluctuations in the light intensity; The camera adjustment unit sets the threshold range of the light intensity, and dynamically adjusts the parameters of the binocular cameras according to the change of the light intensity and the result of the image quality evaluation, including the exposure time, the gain setting, and the white balance parameters; The quality evaluation unit evaluates the quality of the enhanced image, sets a feedback mechanism for the image quality not meeting the standard, and records the camera parameters and the corresponding light intensity data after each adjustment.

[0015] Preferably, the weld seam tracking module is connected with a multi-robot cooperation module through Bluetooth, and the multi-robot cooperation module is used to coordinate the path planning and action timing of multiple robots; The multi-robot cooperation module includes: a space cooperation unit and a task assignment unit. The space cooperation unit is connected with the task assignment unit through Bluetooth, and the vision sensing unit is connected with the space cooperation unit and the task assignment unit through Bluetooth; The space cooperation unit constructs a cooperative kinematic model, calculates the optimal working range of each robot, and coordinates the action timing of multiple robots; The task assignment unit uses a task decomposition and planning algorithm to assign welding tasks, and dynamically adjusts the task assignment scheme according to the priority of the welding tasks and the capabilities of the robots.

[0016] Preferably, the generation of the low-resolution point cloud includes the following steps: Image matching: For the two weld seam images scanned by the binocular cameras, perform feature point matching to match the corresponding pixel points; Disparity calculation: According to the position difference of the matching points and the internal and external parameters of the binocular cameras, calculate the three-dimensional coordinates of each pixel point to generate point cloud data containing depth information.

[0017] Preferably, the generation of the weld seam point cloud includes the following steps: Perform feature extraction on the image frame containing the spot pattern where the laser stripe intersects the weld seam and the background information around the weld seam, and extract the spot pattern of the laser stripe; According to the shape and position information of the spot pattern and the internal and external parameters of the camera, calculate the accurate three-dimensional coordinates of the weld seam to generate weld seam point cloud data.

[0018] Preferably, the area of interest includes: the upper and lower edges of the welding torch, the weld seam area, and the area around the weld seam.

[0019] Compared with the prior art, the beneficial effects of the present invention are: 1. The present invention realizes the automatic positioning function of the weld seam through the design of a weld seam tracking module, solves the problem of weld seam position deviation, improves the welding quality and efficiency of the weld seam robot, and realizes the fusion of active and passive vision; 2. The present invention solves the problem of inaccurate weld seam positioning due to poor image quality during the welding process through the design of a video frame splitting module, improves the accuracy of weld seam positioning and tracking, and improves the accuracy of weld seam recognition; 3. The present invention realizes the segmentation of the weld seam area and other areas in the image through the design of a weld seam defect module, realizes the automatic detection of weld seam defects, improves the efficiency of weld seam defect recognition, and improves the intelligent level of the welding process; 4. The present invention realizes the real-time monitoring and fusion of multi-dimensional data through the design of a sensing fusion module, solves the problem of incomplete data caused by sensor blind spots, improves the integrity and accuracy of data, and improves the accuracy of weld seam recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the weld seam tracking module of the present invention; Figure 2 It is a schematic diagram of the video frame splitting module of the present invention; Figure 3 It is a schematic diagram of the weld seam defect module of the present invention; Figure 4 It is a schematic diagram of the sensing fusion module of the present invention; Figure 5 It is a schematic diagram of the weld seam trimming module of the present invention; Figure 6 It is a schematic diagram of the dynamic compensation module of the present invention; Figure 7 It is a schematic diagram of the weld seam trimming module of the present invention; Figure 8 It is a schematic diagram of the system working process of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0022] Example 1, please refer to Figure 1 and Figure 8, A seam - finding welding robot system based on active and passive vision sensing, including a weld seam tracking module. The weld seam tracking module is used to monitor the weld seam and welding state and adjust the position of the welding torch. The weld seam tracking module includes: a vision sensing unit, a deviation detection unit, and an automatic deviation correction unit. The vision sensing unit is connected to the deviation detection unit via Bluetooth, and the deviation detection unit is connected to the automatic deviation correction unit via Bluetooth. The vision sensing unit scans the weld seam image through a binocular camera to generate a low - resolution point cloud, tracks the rough positioning area, projects a laser stripe in the rough positioning area, collects video images during the welding process, and generates a weld seam point cloud. The deviation detection unit extracts features from the video image data, extracts the upper and lower edge features of the welding torch and the upper and lower groove features of the weld seam, combines the weld seam point cloud data, and calculates the deviation amount between the center line of the welding torch and the center line of the weld seam. The automatic deviation correction unit calculates the direction and distance that the welding torch needs to be adjusted according to the deviation amount, adjusts the position and posture of the welding torch, and controls the deviation between the welding torch and the weld seam within a preset range. The weld seam tracking module is connected to a video frame - splitting module via Bluetooth. The video frame - splitting module is used to split the video images during the welding process into pictures frame by frame for image processing.

[0023] Furthermore, in the passive vision stage, the binocular camera obtains the weld seam image through arc light or captures the weld seam image with natural light. The vision sensing unit collects the weld seam images of the two cameras in the binocular camera respectively, extracts the weld seam images of the two weld seam images, and performs feature point matching. The feature points are pixel points with prominent geometric characteristics such as weld seam edges and corner points. The matching pixel points are found in the two weld seam images, and these pixel points correspond to the same object point in space. According to the triangulation principle, based on the position differences of the matching pixel points and the internal and external parameters of the binocular camera, the three - dimensional coordinates of each pixel point are calculated. During the generation process of the three - dimensional coordinates: Assume that the optical centers of the two cameras of the binocular camera are O1 and O2 respectively, and the corresponding pixel points P1 and P2 of the feature point P in the two images. According to the internal and external parameters of the camera, the equations of the light rays O1P1 and O2P2 are calculated, and the intersection point of the two light rays is solved to obtain the three - dimensional coordinates of the feature point P. Use point cloud generation tools such as PCL, VTK, etc. to convert the three - dimensional coordinate data into a low - resolution point cloud. The vision sensing unit uses a feature extraction algorithm to extract the features of the weld seam in the point cloud data, such as weld seam edges, width, height, etc., so as to determine the approximate position of the weld seam in the point cloud data, and then identify and track the rough positioning area of the weld seam. The rough positioning area is an estimate of the approximate position of the weld seam, providing a range limit for the subsequent precise tracking of the weld seam position. In the active vision stage, a laser emitter or a structured light projection module integrated in the binocular camera projects laser stripes in the rough positioning area. The laser stripes form light bands on the weld surface, which can reflect the precise features of the weld. The visual sensing unit collects video images during the welding process through the binocular camera. The images contain the spot pattern where the laser stripes intersect with the weld, the background information around the weld, etc. The deformation information of the laser stripes reflects the precise position and shape of the weld. The video frame splitting module splits the video images from the visual sensing unit into individual image frames. The visual sensing unit generates weld point cloud data through the same steps as in the passive vision stage based on the image information split and processed by the video frame splitting module. The weld point cloud data reflects the three-dimensional morphology information of the weld. The visual sensing unit registers the generated low-resolution point cloud with the high-precision weld point cloud, uses the ICP algorithm for registration, iterates the closest points, adjusts the pose of the point cloud, continuously reduces the error between the two point clouds, and generates unified high-precision point cloud data. By fusing the data of active vision and passive vision, more precise and stable weld point clouds are generated. In addition, while the binocular camera collects the weld video images, the visual sensing unit uses a high-speed camera to collect welding pool images to obtain information about the welding torch and the welding pool, thereby realizing the dynamic monitoring of the welding process in passive vision. Through passive vision, the preliminary positioning of the weld is provided, and active vision performs precise measurement by projecting laser stripes.

[0024] The deviation detection unit receives the video image data collected by the visual sensing unit, extracts the features of the image data through image processing algorithms, and identifies the upper and lower edge positions of the welding torch in the image. At the same time, it provides a reference for subsequent deviation calculation. The groove shape, size, etc. of the weld are identified by extracting the weld groove features through edge detection methods. The deviation detection unit integrates the weld image features and point cloud features, and generates a three-dimensional model of the weld through a three-dimensional reconstruction algorithm. The shortest distance from each point on the center line of the welding torch to the center line of the weld is accurately calculated in real time through the three-dimensional model, and the maximum value is taken as the distance deviation. The angle between the center line of the welding torch and the center line of the weld is calculated as the angle deviation. The automatic deviation correction unit receives the real-time deviation amount calculated by the deviation detection unit. The automatic deviation correction unit calculates the XYZ three-axis translation amounts that the welding torch needs to adjust according to the real-time deviation amount, combines the rotation axis, calculates the inclination angle and position of the welding torch, and the automatic deviation correction unit adjusts the position and pose of the welding torch in real time according to the calculation results of the welding torch to ensure that the deviation between the welding torch and the weld is controlled within the preset range.

[0025] Example 2, please refer to Figure 2 and Figure 8, A seam - finding welding robot system based on active and passive vision sensing. The seam - tracking module is connected to a video frame - splitting module via Bluetooth. The video frame - splitting module is used to split the video images during the welding process into individual frames one by one for image processing. The video frame - splitting module includes: a frame - splitting processing unit, a field - of - view correction unit, and an image enhancement unit. The frame - splitting processing unit is connected to the field - of - view correction unit via Bluetooth, the field - of - view correction unit is connected to the image enhancement unit via Bluetooth, and the vision sensing unit is connected to the frame - splitting processing unit via Bluetooth. The frame - splitting processing unit uses a video frame - splitting algorithm to perform frame - splitting processing on the video images during the welding process, splitting the video stream into individual image frames one by one, and storing the image frames in the database in sequence. The field - of - view correction unit uses an image - processing algorithm to identify the contours in the image frame, select the region of interest, calculate the tilt angle of the image, and use a geometric transformation algorithm to rotate and correct the image to restore the horizontal and vertical states of the region of interest. The image enhancement unit performs histogram analysis on the image frame, obtains the gray - value range of the region of interest, and uses an image - enhancement algorithm to perform enhancement processing on the image frame.

[0026] Furthermore, the frame - splitting processing unit uses a video frame - splitting algorithm to convert the continuous video stream into image frames, performs fine splitting on the video data generated during the welding process to ensure that each frame of the image is accurately extracted. In addition, the frame - splitting processing unit introduces parallel - processing technology, uses a multi - core CPU or GPU for acceleration, thereby achieving fast frame - splitting processing of the video stream, and uses distributed storage to store the image frames in the database in order and number them. The field - of - view correction unit extracts the image frames from the database and analyzes each frame of the image output and stored by the frame - splitting processing unit through an image - processing algorithm to accurately identify the contour features in the image. At the same time, the field - of - view correction unit automatically selects the region of interest on the image. The region of interest usually includes key information such as the upper and lower edges of the welding torch, the weld region, and the area around the weld, including the welding torch and the weld. The field - of - view correction unit detects the geometric features within the region of interest and performs mathematical calculations through the least - squares method to obtain the tilt angle of the image, and then uses a geometric transformation algorithm to rotate and correct the image to restore the horizontal and vertical states of the region of interest. The image enhancement unit performs histogram analysis on the image frames that have undergone frame - splitting and correction processing, obtains the gray - value range of the region of interest, and performs enhancement processing on the image according to the distribution of the gray values to make the gray - value distribution more uniform. In addition, the image enhancement unit combines multiple technologies such as adaptive histogram equalization and Laplacian pyramid enhancement. When high - brightness arc light, low contrast, etc. occur in the region of interest in the weld image, the image enhancement unit performs corresponding image - enhancement processing, such as enhancing the contrast of the image and adjusting the brightness, so as to improve the accuracy of the key features such as the welding torch and the weld when the seam - tracking module extracts them from the image.

[0027] Example 3, please refer toFigure 3 and Figure 8 , a seam - finding welding robot system based on active - passive vision sensing. The seam - tracking module is connected to a weld - defect module via Bluetooth. The weld - defect module is used for detecting weld defects. The weld - defect module includes: a region - extraction unit, a defect - segmentation unit, and a defect - identification unit. The region - extraction unit is connected to the defect - segmentation unit via Bluetooth, and the defect - segmentation unit is connected to the defect - identification unit via Bluetooth. The region - extraction unit calculates the curvature value of each data in the weld point cloud, initially segments the weld region and other regions, performs point - cloud slicing on other regions, defines the weld region, and combines to obtain the complete weld - region point cloud. The defect - segmentation unit divides the weld - region point cloud into grids, compares the number of points in the grid with a preset threshold, segments the burn - through region and the planar region, and uses the region - growing segmentation algorithm to process the planar region and segment the defect region. The defect - identification unit extracts the point - cloud features of the defect region, constructs an identification and classification model for weld defects, and automatically identifies and classifies the defects, including weld beads, depressions, welding slag, and pores.

[0028] Furthermore, the region - extraction unit receives the weld - point - cloud data generated by the seam - tracking module, calculates the curvature value of each data in the weld point cloud, and evaluates the curvature value of each data point in the weld point cloud. The curvature reflects the geometric features of the weld - point - cloud surface. The high - curvature region corresponds to key parts such as the edges, turns, or defects of the weld. The region - extraction unit identifies the boundary between the weld region and the background region by setting a curvature threshold. For example, in the weld point cloud, the curvature value of the weld centerline or edge is high, while the curvature value of the flat background region is low. By setting the threshold, the region - extraction unit can filter out the weld region to be processed, thus initially separating the weld region and other regions. After obtaining the initial segmentation result, the region - extraction unit uses the point - cloud slicing technology. By using slicing software to cut the weld point cloud of other regions, two - dimensional slices are obtained. The sliced point - cloud data is subjected to detailed feature extraction, such as further edge detection, etc., to obtain the fine - structure features of other regions, thereby further defining the scope of the weld region. By combining the point - cloud data, the complete weld - region point cloud is obtained. Before the region - extraction unit performs slicing fitting, the approximate weld region is initially segmented, the other regions outside the approximate weld region are sliced, the weld region is further defined within other regions, and by combining the initially segmented weld region and the further - defined weld region, while obtaining the complete weld region, the processing efficiency of the region - extraction unit is improved; The defect segmentation unit uses the grid division technology to divide the weld seam area point cloud generated by the area extraction unit into multiple small grids. By calculating the number of points in each grid and comparing it with a preset threshold, when the number of points in a grid is lower than the threshold, the defect segmentation unit marks it as a burn-through area, and the others are marked as flat areas. For the flat areas, the defect segmentation unit uses the region growing segmentation algorithm for processing. The defect segmentation unit selects multiple seed points and expands the region according to the similarity of the point cloud data, adding the points that meet the similarity conditions to the expanded region and updating the boundary point set, so as to separate each expanded region in the flat area and separate the point set expanded regions with significant differences in normal direction, geometric shape, etc. from the surrounding points, and marking these expanded regions as defect regions; the defect recognition unit extracts the features of the defect region point cloud separated by the defect segmentation unit, constructs an identification and classification model for weld defects, and identifies the feature distribution and classification rules of the weld defect region through the labeled weld defect sample data. The defect recognition unit matches the features in the model with the features of the defect region point cloud to identify the defect types in the weld region, such as weld beads, depressions, welding slag, and pores, etc., and classifies them according to the types of weld defects.

[0029] Example 4, please refer to Figure 1 、 Figure 2 、 Figure 4 and Figure 8 , a seam-tracking robot system based on active and passive vision sensing. The weld seam tracking module is connected with a sensing fusion module through Bluetooth. The sensing fusion module is used to integrate different sensor data, adjust the parameters and working modes of the sensors; the sensing fusion module includes: a multi-dimensional fusion unit, a blind area compensation unit, and a dynamic cooperation unit. The multi-dimensional fusion unit is connected with the blind area compensation unit through Bluetooth, and the multi-dimensional fusion unit is connected with the dynamic cooperation unit through Bluetooth; the multi-dimensional fusion unit collects the real-time monitoring data of the sensor network, including a force sensor, an arc sensor, and a temperature sensor, combines the image data in the welding process, and fuses the multi-dimensional data using a data fusion algorithm; the blind area compensation unit performs pattern recognition based on the multi-dimensional data, identifies the blind areas of the sensors, determines the missing situation of the multi-dimensional data, and uses the multi-dimensional data to compensate the blind area information of other sensors; the dynamic cooperation unit analyzes the multi-dimensional data in real time, identifies the changes in the position, shape, and welding environment of the weld seam, and dynamically adjusts the working parameters of the sensor network and the welding parameters according to the specific requirements of the welding task; It includes a weld seam tracking module which is used to monitor the weld seam and welding state and adjust the position of the welding torch. The weld seam tracking module includes a visual sensing unit, a deviation detection unit and an automatic deviation correction unit. The visual sensing unit is connected to the deviation detection unit via Bluetooth, and the deviation detection unit is connected to the automatic deviation correction unit via Bluetooth. The weld seam tracking module is connected to a video frame splitting module via Bluetooth. The video frame splitting module is used to split the video images during the welding process into individual frames for image processing. The video frame splitting module includes a frame splitting processing unit, a field of view correction unit and an image enhancement unit. The frame splitting processing unit is connected to the field of view correction unit via Bluetooth, the field of view correction unit is connected to the image enhancement unit via Bluetooth, and the visual sensing unit is connected to the frame splitting processing unit via Bluetooth.

[0030] Furthermore, the multi-dimensional fusion unit realizes the acquisition of monitoring data from various sensors such as force sensors, arc sensors, and temperature sensors. Through data fusion algorithms such as Kalman filtering, particle filtering, neural networks, etc., the multi-dimensional data are fused. The data from different sensors are aligned in time and space, weighted, and optimally combined to integrate the real-time monitoring data of multiple sensors, providing the system with a comprehensive perception of the welding process. For example, the force sensor data reflects the contact force between the welding torch and the weld seam, the arc sensor data reflects the stability of the arc and the welding quality, and the temperature sensor data reflects the thermal distribution in the welding area. In addition, the multi-dimensional fusion unit supports the fusion of multi-source heterogeneous data, combining the active vision data and passive vision data of the visual sensing unit. The blind area compensation unit analyzes the multi-dimensional data fused by the multi-dimensional fusion unit, identifies the data missing due to the complex welding environment and the limitations of the sensors themselves, and identifies the data missing or abnormal situations caused by certain areas or angles in the welding process that cannot be fully monitored by a single sensor. The blind area compensation unit identifies the blind area information based on the data missing or abnormal situations, calls the relevant data of other sensors, and complements the blind area information through methods such as data interpolation and fusion. For example, when there is smoke and dust occlusion in the video image of the visual sensing unit, the blind area compensation unit uses the arc current waveform of the arc sensor data and the temperature field distribution of the temperature sensor to generate point cloud complementary data. In addition, the blind area compensation unit can also dynamically adjust the weight parameters of each sensor through fuzzy logic. The dynamic cooperation unit analyzes the multi-dimensional data processed by the blind area compensation unit in real time, identifies the position and shape of the weld seam and the changes in the welding environment, and dynamically adjusts the welding parameters according to the specific requirements of the welding task, such as adjusting the swing amplitude of the welding torch, the welding speed, the current, etc., and adjusting the working parameters of the sensor network, such as setting the corresponding sensor as the dominant mode according to the weight parameters of each sensor and turning off redundant sensors in the stable welding section. In addition, according to the welding parameters adjusted by the dynamic cooperation unit, the visual sensing unit in the weld tracking module dynamically adjusts the frame rate of the entire video acquisition of the binocular camera to avoid video blurring during high-speed welding, so as to ensure that the frame splitting processing unit of the subsequent video frame splitting module obtains clear images. At the same time, the frame splitting processing unit combines with the sensing fusion module to add real-time welding parameters and environmental data to each split frame image, generating spatio-temporally aligned multi-dimensional data for subsequent weld defects.

[0031] Example 5, please refer to Figure 5 、 Figure 7 and Figure 8 , a seam tracking robot system based on active and passive visual sensing. The weld defect module is connected to a weld repair module via Bluetooth. The weld repair module is used to process weld defects. The weld repair module includes: a repair planning unit and a trajectory planning unit. The repair planning unit is connected to the trajectory planning unit via Bluetooth, and the repair planning unit is connected to a defect recognition unit via Bluetooth. The repair planning unit collects the results of weld defect recognition and classification, including defect type, defect location, and defect size information, and combines the physical properties of the welding material and the welding process parameters to formulate a weld repair strategy, including the selection of repair tools, the planning of machining swing angles, and the planning of feed speeds. The trajectory planning unit calculates and optimizes the collision-free machining path of the repair tool according to the weld repair strategy and the weld defect data using a path planning algorithm. The weld tracking module is connected to a weld defect module via Bluetooth. The weld defect module is used for defect detection of the weld. The weld defect module includes: a region extraction unit, a defect segmentation unit, and a defect recognition unit. The region extraction unit is connected to the defect segmentation unit via Bluetooth, and the defect segmentation unit is connected to the defect recognition unit via Bluetooth. The weld tracking module is connected to a multi-robot cooperation module via Bluetooth. The multi-robot cooperation module is used to coordinate the path planning and action timing of multiple robots. The multi-robot cooperation module includes: a space cooperation unit and a task assignment unit. The space cooperation unit is connected to the task assignment unit via Bluetooth, and the visual sensing unit is connected to the space cooperation unit and the task assignment unit via Bluetooth. The space cooperation unit constructs a cooperative kinematic model, calculates the optimal working range of each robot, and coordinates the action timing of multiple robots. The task assignment unit uses a task decomposition and planning algorithm to assign welding tasks and dynamically adjusts the task assignment scheme according to the priority of the welding tasks and the capabilities of the robots.

[0032] Furthermore, the trimming planning unit receives the results of weld defect identification and classification analyzed by the weld defect module, including defect type, defect location, and defect size information. It constructs a welding material database in combination with the physical properties of the welding materials, and formulates a weld trimming strategy in combination with the welding process parameters and weld defect information. For example, it determines what kind of trimming tool is required for the corresponding defect type, predicts the surface roughness after weld trimming according to the robot learning model, simulates the situations of different machining swing angles and feed rates, and selects the optimal parameters. The trajectory planning unit constructs a trimming tool kinematic model according to the weld trimming strategy formulated by the trimming planning unit and the weld defect data from the weld defect module, solves the optimal trajectory using the ant colony algorithm, and generates a smooth trimming path through the curvature continuous interpolation algorithm to complete the optimization of the machining path. During the process of the system controlling the trimming robot to trim the weld defect according to the machining path generated by the trajectory planning unit, the spatial coordination unit couples the models of the weld robot and the trimming robot, constructs a cooperative kinematic model, calculates the optimal working range of each robot according to the predicted action time sequence and spatial interference of multiple robots in the model, coordinates the action time sequence of multiple robots, and controls multiple robots to work cooperatively while avoiding robot collisions, so as to realize multiple robots performing welding tasks simultaneously or performing welding tasks and trimming tasks simultaneously. The task allocation unit uses the task decomposition and planning algorithm to decompose an overall welding task into multiple subtasks, constructs a task priority model, assigns the priorities of the subtasks, and dynamically adjusts the task allocation scheme according to the priority of the welding task and the capabilities of the robots.

[0033] Example 6, please refer to Figure 6 and Figure 8 , a seam-tracking weld robot system based on active and passive vision sensing. The image enhancement unit is connected with a dynamic compensation module through Bluetooth. The dynamic compensation module is used to monitor the light intensity of the welding area in real time and adjust the parameters of the binocular camera. The dynamic compensation module includes a light intensity monitoring unit, a camera adjustment unit, and a quality evaluation unit. The light intensity monitoring unit is connected with the camera adjustment unit through Bluetooth, the camera adjustment unit is connected with the quality evaluation unit through Bluetooth, and the image enhancement unit is connected with the quality evaluation unit through Bluetooth. The light intensity monitoring unit monitors the light intensity of the welding area in real time through the light intensity sensor carried by the binocular camera, analyzes the change trend of the light intensity, and identifies the fluctuation situation of the light intensity. The camera adjustment unit sets the threshold range of the light intensity, and dynamically adjusts the parameters of the binocular camera according to the change situation of the light intensity and the image quality evaluation result, including exposure time, gain setting, and white balance parameters. The quality evaluation unit evaluates the quality of the enhanced image, sets a feedback mechanism for the image quality not meeting the standard, and records the camera parameters and the corresponding light intensity data after each adjustment.

[0034] Furthermore, during the welding process, the light intensity in the welding area is affected by factors such as arc light, the light and shadow changes in the surrounding environment, and the laser stripes projected on the weld area. The light intensity monitoring unit continuously collects the light intensity monitoring data of the light intensity sensor. By collecting continuous monitoring data and performing data analysis, the light intensity monitoring unit obtains the change trend of the light intensity. The light intensity monitoring unit receives the image data processed by the image enhancement unit, and combines the image data with the light intensity at the corresponding time point to identify the light fluctuations that have a negative impact on the welding image quality. The camera adjustment unit receives the light fluctuation data that affects the image quality analyzed by the light intensity monitoring unit, combines the historical light intensity data and the corresponding image quality data, and sets the threshold range of the light intensity. When the camera adjustment unit detects that the light intensity exceeds or falls below these thresholds, the camera adjustment unit dynamically adjusts the parameters of the binocular camera according to the change of the light intensity. After triggering the feedback mechanism, the camera adjustment unit fine-tunes the parameters of the binocular camera according to the image quality evaluation result until the best image effect is achieved. The quality evaluation unit evaluates the quality of the images processed by the image enhancement unit, sets up a feedback mechanism when the image quality does not meet the standard. When the image quality does not meet the standard, the quality evaluation unit notifies the camera adjustment unit to adjust the parameters of the binocular camera. At the same time, the quality evaluation unit records the camera parameters and the corresponding light intensity data after each adjustment, providing a data basis for subsequent welding quality analysis.

[0035] Working principle: During the welding process, the weld seam tracking module continuously collects the video during welding. The video frame splitting module splits the video stream from the weld seam tracking module into individual image frames and corrects and enhances the images. Based on the images processed by the video frame splitting module, the weld seam tracking module generates weld seam point cloud data; The sensing fusion module maps the multi-dimensional data collected by the sensor network into the images split by the video frame splitting module. The sensing fusion module integrates the multi-dimensional data and the image data, compensates for the blind area information, generates point cloud completion data, and combines the point cloud completion data and the weld seam point cloud data to obtain the complete point cloud data; The weld seam tracking module calculates the deviation between the center line of the welding torch and the center line of the weld seam based on the point cloud data, and adjusts the position and posture of the welding torch. At the same time, the weld defect module calculates the curvature value of the point cloud data to identify weld defects.

[0036] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-described exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention. Any reference signs in the claims should not be construed as limiting the claims involved.

Claims

1. A seam - seeking welding robot system based on active and passive vision sensing, characterized in that: It includes a weld seam tracking module, which is used to monitor the weld seam and welding state and adjust the position of the welding torch; The weld seam tracking module includes: a visual sensing unit, a deviation detection unit, and an automatic deviation correction unit. The visual sensing unit is connected to the deviation detection unit via Bluetooth, and the deviation detection unit is connected to the automatic deviation correction unit via Bluetooth; The visual sensing unit scans the weld seam image through a binocular camera to generate a low-resolution point cloud, tracks the rough positioning area, projects a laser stripe in the rough positioning area, collects the video image during the welding process, and generates a weld seam point cloud; The deviation detection unit extracts features from the video image data, extracts the upper and lower edge features of the welding torch and the upper and lower groove features of the weld seam, and combines the weld seam point cloud data to calculate the deviation amount between the center line of the welding torch and the center line of the weld seam; The automatic deviation correction unit calculates the direction and distance that the welding torch needs to be adjusted according to the deviation amount, adjusts the position and posture of the welding torch, and controls the deviation between the welding torch and the weld seam within a preset range.

2. The seam - seeking welding robot system based on active - passive vision sensing according to claim 1, wherein: The weld seam tracking module is connected to a video frame splitting module via Bluetooth. The video frame splitting module is used to split the video image during the welding process into pictures frame by frame for image processing; The video frame splitting module includes: a frame splitting processing unit, a field of view correction unit, and an image enhancement unit. The frame splitting processing unit is connected to the field of view correction unit via Bluetooth, the field of view correction unit is connected to the image enhancement unit via Bluetooth, and the visual sensing unit is connected to the frame splitting processing unit via Bluetooth; The frame splitting processing unit uses a video frame splitting algorithm to perform frame splitting processing on the video image during the welding process, splits the video stream into individual image frames frame by frame, and stores the image frames in the database in sequence; The field of view correction unit uses an image processing algorithm to identify the contour in the image frame, selects the interested area, calculates the tilt angle of the image, and uses a geometric transformation algorithm to rotate and correct the image to restore the horizontal and vertical states of the interested area; The image enhancement unit performs histogram analysis on the image frame, obtains the gray value range of the interested area, and uses an image enhancement algorithm to perform enhancement processing on the image frame.

3. The seam - seeking welding robot system based on active and passive vision sensing according to claim 1, characterized in that: The weld seam tracking module is connected to a weld seam defect module via Bluetooth. The weld seam defect module is used for defect detection of the weld seam; The weld seam defect module includes: a region extraction unit, a defect segmentation unit, and a defect recognition unit. The region extraction unit is connected to the defect segmentation unit via Bluetooth, and the defect segmentation unit is connected to the defect recognition unit via Bluetooth; The region extraction unit calculates the curvature value of each data in the weld seam point cloud, initially segments the weld seam region and other regions, performs point cloud slicing processing on other regions, defines the weld seam region, and combines to obtain the complete weld seam region point cloud; The defect segmentation unit divides the weld seam region point cloud into grids, compares the number of points in the grid with a preset threshold, segments the burn-through region and the plane region, and uses a region growing segmentation algorithm to process the plane region to segment the defect region; The defect recognition unit extracts the point cloud features of the defect region, constructs an identification and classification model for weld seam defects, and automatically identifies and classifies the defects, including weld beads, depressions, welding slag, and pores.

4. A seam - seeking welding robot system based on active - passive vision sensing according to claim 1, characterized in that: The weld seam tracking module is connected to a sensing fusion module via Bluetooth. The sensing fusion module is used to integrate data from different sensors, and adjust the parameters and working modes of the sensors. The sensing fusion module includes: a multi-dimensional fusion unit, a blind area compensation unit, and a dynamic cooperation unit. The multi-dimensional fusion unit is connected to the blind area compensation unit via Bluetooth, and the multi-dimensional fusion unit is connected to the dynamic cooperation unit via Bluetooth. The multi-dimensional fusion unit collects real-time monitoring data from the sensor network, including a force sensor, an arc sensor, and a temperature sensor, combines the image data during the welding process, and fuses the multi-dimensional data using a data fusion algorithm. The blind area compensation unit performs pattern recognition based on the multi-dimensional data, identifies the blind areas of the sensors, determines the missing situation of the multi-dimensional data, and uses the multi-dimensional data to compensate for the blind area information of other sensors. The dynamic cooperation unit analyzes the multi-dimensional data in real time, identifies the changes in the position, shape of the weld seam and the welding environment, and dynamically adjusts the working parameters of the sensor network and the welding parameters according to the specific requirements of the welding task.

5. The seam - searching welding robot system based on active - passive vision sensing according to claim 3, characterized in that: The weld defect module is connected to a weld repair module via Bluetooth. The weld repair module is used to process weld defects. The weld repair module includes: a repair planning unit, a trajectory planning unit. The repair planning unit is connected to the trajectory planning unit via Bluetooth, and the repair planning unit is connected to a defect identification unit via Bluetooth. The repair planning unit collects the results of weld defect identification and classification, including defect type, defect location, and defect size information, combines the physical properties of the welding material and the welding process parameters, and formulates a weld repair strategy, including the selection of repair tools, the planning of machining swing angles, and the planning of feed speeds. The trajectory planning unit calculates and optimizes the collision-free machining path of the repair tool according to the weld repair strategy and the weld defect data using a path planning algorithm.

6. The seam - searching welding robot system based on active - passive vision sensing according to claim 2, wherein: The image enhancement unit is connected to a dynamic compensation module via Bluetooth. The dynamic compensation module is used to monitor the light intensity of the welding area in real time and adjust the parameters of the binocular camera. The dynamic compensation module includes a light monitoring unit, a camera adjustment unit, and a quality evaluation unit. The light monitoring unit is connected to the camera adjustment unit via Bluetooth, the camera adjustment unit is connected to the quality evaluation unit via Bluetooth, and the image enhancement unit is connected to the quality evaluation unit via Bluetooth. The light monitoring unit monitors the light intensity of the welding area in real time through the light intensity sensor carried by the binocular camera, analyzes the change trend of the light intensity, and identifies the fluctuation of the light intensity. The camera adjustment unit sets the threshold range of the light intensity, and dynamically adjusts the parameters of the binocular camera according to the change of the light intensity and the image quality evaluation result, including the exposure time, gain setting, and white balance parameter. The quality evaluation unit evaluates the quality of the enhanced image, sets a feedback mechanism for the image quality not meeting the standard, and records the camera parameters and the corresponding light intensity data after each adjustment.

7. A seam - seeking welding robot system based on active and passive vision sensing according to claim 1, wherein: The weld seam tracking module is connected to a multi-robot cooperation module via Bluetooth. The multi-robot cooperation module is used to coordinate the path planning and action timing of multiple robots. The multi-robot cooperation module includes: a spatial cooperation unit and a task allocation unit. The spatial cooperation unit is connected to the task allocation unit via Bluetooth, and the vision sensing unit is connected to the spatial cooperation unit and the task allocation unit via Bluetooth; The spatial cooperation unit constructs a cooperative kinematic model, calculates the optimal working range of each robot, and coordinates the action timings of multiple robots; The task allocation unit uses a task decomposition and planning algorithm to allocate welding tasks and dynamically adjusts the task allocation scheme according to the priorities of the welding tasks and the capabilities of the robots.

8. A seam - seeking welding robot system based on active - passive vision sensing according to claim 1, characterized in that: The generation of the low-resolution point cloud includes the following steps: Image matching: For two weld images scanned by a binocular camera, perform feature point matching to match the corresponding pixel points; Disparity calculation: According to the position differences of the matching points and the internal and external parameters of the binocular camera, calculate the three-dimensional coordinates of each pixel point to generate point cloud data containing depth information.

9. The seam - seeking welding robot system based on active - passive vision sensing according to claim 1, characterized in that: The generation of the weld point cloud includes the following steps: Extract features from an image frame containing the spot pattern where the laser stripe intersects the weld and the background information around the weld, and extract the spot pattern of the laser stripe; According to the shape and position information of the spot pattern and the internal and external parameters of the camera, calculate the accurate three-dimensional coordinates of the weld to generate weld point cloud data.

10. A seam-finding welding robot system based on active and passive vision sensing according to claim 2, characterized in that: The area of interest includes: the upper and lower edges of the welding torch, the weld area, and the area around the weld.

Images