A seam-finding robot system based on active and passive vision

Abstract

The application discloses a kind of based on active and passive visual sensing's location weld robot system, it is related to robot welding technical field, including weld tracking module, the weld tracking module is used to monitor weld and welding state, adjusts welding torch position;The weld tracking module includes: visual sensing unit, deviation detection unit and automatic deviation correction unit, the visual sensing unit is connected with deviation detection unit by Bluetooth, and the deviation detection unit is connected with automatic deviation correction unit by Bluetooth.The application is designed with weld tracking module, realizes the automatic positioning function of weld, solves the problem of weld position deviation, improves the welding quality and efficiency of weld robot, realizes the fusion of active and passive vision.

Description

Technical Field

[0001] This invention relates to the field of robotic welding technology, specifically to a positioning welding robot system based on active and passive vision sensing. Background Technology

[0002] As an important component of modern industrial automation, welding robot systems are widely used in the automotive, shipbuilding, aerospace, and steel structure industries. Compared to traditional manual welding, welding robot systems offer advantages such as high precision and high efficiency, reducing labor costs and improving operational safety.

[0003] Traditional welding robot systems typically weld along pre-programmed fixed trajectories. When subjected to factors such as workpiece deformation, assembly errors, or environmental interference, they cannot make adaptive adjustments, leading to weld position deviations. This affects welding quality and increases rework rates and production costs.

[0004] Patent CN114952098B discloses an intelligent weld seam positioning system and welding robot based on machine vision guidance. The above patent enables further confirmation of weld seam position, improves weld seam positioning efficiency, and enables quality inspection after welding.

[0005] The aforementioned patent determines the specific location of the weld, solving the problem of inaccurate weld positioning by existing weld positioning robots. However, there is still room for optimization in terms of weld position offset. This application calculates the deviation between the welding torch and the weld to solve the problem of weld position offset.

[0006] Therefore, this application proposes a positioning welding robot system based on active and passive vision sensing to realize automatic control of the deviation between the welding torch and the weld. Summary of the Invention

[0007] The purpose of this invention is to provide a positioning welding robot system based on active and passive vision sensing to solve the technical problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a positioning welding robot system based on active and passive vision sensing, including a weld tracking module, wherein the weld tracking module is used to monitor the weld and welding status and adjust the position of the welding torch; The weld seam tracking module includes: a vision sensing unit, a deviation detection unit, and an automatic correction unit. The vision sensing unit is connected to the deviation detection unit via Bluetooth, and the deviation detection unit is connected to the automatic correction unit via Bluetooth. The visual sensing unit scans the weld seam image with a binocular camera to generate a low-resolution point cloud, tracks the coarse positioning area, projects laser stripes in the coarse positioning area, and collects video images of the welding process to generate the weld seam point cloud. The deviation detection unit extracts features from the video image data, extracts the upper and lower edge features of the welding torch and the upper and lower bevel features of the weld, and calculates the deviation between the center line of the welding torch and the center line of the weld by combining the weld point cloud data. The automatic correction unit calculates the direction and distance that the welding torch needs to be adjusted based on the deviation, adjusts the position and posture of the welding torch, and controls the deviation between the welding torch and the weld within a preset range.

[0009] Preferably, the weld seam tracking module is connected to a video frame splitting module via Bluetooth. The video frame splitting module is used to split the video images during the welding process into images frame by frame for image processing.

[0010] The video frame splitting module includes: a frame splitting processing unit, a field of view correction unit, and an image enhancement unit. The frame splitting processing unit is connected to the field of view correction unit via Bluetooth, the field of view correction unit is connected to the image enhancement unit via Bluetooth, and the visual sensing unit is connected to the frame splitting processing unit via Bluetooth. The frame splitting processing unit uses a video frame splitting algorithm to split the video images during the welding process into individual image frames, and stores the image frames in the database in sequence. The field correction unit uses image processing algorithms to identify contours in image frames, select regions of interest, calculate the tilt angle of the image, and use geometric transformation algorithms to rotate and correct the image to restore the horizontal and vertical states of the regions of interest. The image enhancement unit performs histogram analysis on the image frame to obtain the gray value range of the region of interest, and then uses an image enhancement algorithm to enhance the image frame.

[0011] Preferably, the weld seam tracking module is connected to a weld seam defect module via Bluetooth, and the weld seam defect module is used for weld seam defect detection. The weld defect module includes: a region extraction unit, a defect segmentation unit, and a defect recognition unit. The region extraction unit is connected to the defect segmentation unit via Bluetooth, and the defect segmentation unit is connected to the defect recognition unit via Bluetooth. The region extraction unit calculates the curvature value of each data point in the weld point cloud, initially segments the weld region and other regions, performs point cloud slicing on the other regions, defines the weld region, and merges them to obtain the complete weld region point cloud. The defect segmentation unit divides the point cloud of the weld area into a grid, compares the number of points in the grid with a preset threshold, and segments the burn-through area and the planar area. The planar area is processed by the region growth segmentation algorithm to segment the defect area. The defect identification unit extracts the point cloud features of the defect area, constructs a model for identifying and classifying weld defects, and automatically identifies and classifies defects, including weld beads, depressions, weld slag, and porosity.

[0012] Preferably, the weld seam tracking module is connected to a sensor fusion module via Bluetooth. The sensor fusion module is used to integrate data from different sensors and adjust the parameters and working modes of the sensors. The sensor fusion module includes: a multi-dimensional fusion unit, a blind spot compensation unit, and a dynamic coordination unit. The multi-dimensional fusion unit is connected to the blind spot compensation unit via Bluetooth, and the multi-dimensional fusion unit is connected to the dynamic coordination unit via Bluetooth. The multi-dimensional fusion unit collects real-time monitoring data from the sensor network, including force sensors, arc sensors, and temperature sensors. Combined with image data from the welding process, the multi-dimensional data is fused using a data fusion algorithm. The blind spot compensation unit performs pattern recognition based on multidimensional data to identify the blind spots of the sensors, determine the missing multidimensional data, and use the multidimensional data to compensate for the blind spot information of other sensors. The dynamic collaborative unit analyzes multi-dimensional data in real time, identifies the location, shape, and changes in the welding environment of the weld, and dynamically adjusts the working parameters of the sensor network and the welding parameters according to the specific requirements of the welding task.

[0013] Preferably, the weld defect module is connected to a weld repair module via Bluetooth, and the weld repair module is used to process weld defects. The weld seam trimming module includes: a trimming planning unit and a trajectory planning unit. The trimming planning unit is connected to the trajectory planning unit via Bluetooth, and the trimming planning unit is connected to the defect recognition unit via Bluetooth. The finishing planning unit collects the results of weld defect identification and classification, including defect type, defect location, and defect size information. Combined with the physical properties of the welding material and welding process parameters, it formulates weld finishing strategies, including the selection of finishing tools, the planning of machining angle, and the planning of feed speed. Based on the weld repair strategy and weld defect data, the trajectory planning unit uses a path planning algorithm to calculate and optimize the collision-free processing path of the repair tool.

[0014] Preferably, the image enhancement unit is connected to a dynamic compensation module via Bluetooth. The dynamic compensation module is used to monitor the light intensity of the welding area in real time and adjust the parameters of the binocular camera. The dynamic compensation module includes a light monitoring unit, a camera adjustment unit, and a quality assessment unit. The light monitoring unit is connected to the camera adjustment unit via Bluetooth, the camera adjustment unit is connected to the quality assessment unit via Bluetooth, and the image enhancement unit is connected to the quality assessment unit via Bluetooth. The illumination monitoring unit monitors the illumination intensity of the welding area in real time through the illumination intensity sensor mounted on the binocular camera, analyzes the trend of illumination intensity changes, and identifies the fluctuations in illumination intensity. The camera adjustment unit sets a threshold range for light intensity and dynamically adjusts the parameters of the binocular camera, including exposure time, gain settings, and white balance parameters, based on changes in light intensity and image quality assessment results. The quality assessment unit evaluates the quality of the enhanced image, sets up a feedback mechanism for images whose quality does not meet the standard, and records the camera parameters and corresponding illumination intensity data after each adjustment.

[0015] Preferably, the weld seam tracking module is connected to a multi-machine collaboration module via Bluetooth, and the multi-machine collaboration module is used to coordinate the path planning and action timing of multiple robots; The multi-machine collaboration module includes: a spatial collaboration unit and a task allocation unit. The spatial collaboration unit is connected to the task allocation unit via Bluetooth, and the visual sensing unit is connected to both the spatial collaboration unit and the task allocation unit via Bluetooth. The spatial collaborative unit constructs a collaborative kinematics model, calculates the optimal working range of each robot, and coordinates the action sequence of multiple robots. The task allocation unit uses task decomposition and planning algorithms to allocate welding tasks and dynamically adjusts the task allocation scheme based on the priority of the welding tasks and the capabilities of the robot.

[0016] Preferably, the generation of the low-resolution point cloud includes the following steps: Image matching: Feature point matching is performed on two weld seam images scanned by a binocular camera to match the corresponding pixels; Parallax calculation: Based on the positional differences of the matching points and the intrinsic and extrinsic parameters of the binocular camera, the three-dimensional coordinates of each pixel are calculated to generate point cloud data containing depth information.

[0017] Preferably, the generation of the weld point cloud includes the following steps: Feature extraction is performed on image frames containing the spot pattern of laser stripes intersecting with the weld and background information around the weld to extract the spot pattern of the laser stripes; Based on the shape and position information of the light spot pattern and the camera's internal and external parameters, the precise three-dimensional coordinates of the weld are calculated, and weld point cloud data is generated.

[0018] Preferably, the area of ​​interest includes: the upper and lower edges of the welding torch, the weld area, and the area surrounding the weld.

[0019] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention, by designing a weld seam tracking module, realizes the automatic positioning function of weld seams, solves the problem of weld seam position offset, improves the welding quality and efficiency of welding robots, and achieves the integration of active and passive vision; 2. This invention, by incorporating a video frame splitting module, solves the problems of poor image quality and inaccurate weld positioning during the welding process, thereby improving the accuracy of weld positioning and tracking, and enhancing the accuracy of weld recognition. 3. This invention, by designing a weld defect module, realizes the segmentation of the weld area and other areas in the image, achieves automatic detection of weld defects, improves the efficiency of weld defect identification, and enhances the intelligence level of the welding process; 4. This invention, through the design of a sensor fusion module, realizes real-time monitoring and fusion of multi-dimensional data, solves the problem of incomplete data caused by sensor blind spots, improves data integrity and accuracy, and enhances the precision of weld seam identification. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the weld seam tracking module of the present invention; Figure 2 This is a schematic diagram of the video frame splitting module of the present invention; Figure 3 This is a schematic diagram of the weld defect module of the present invention; Figure 4 This is a schematic diagram of the sensor fusion module of the present invention; Figure 5 This is a schematic diagram of the weld seam trimming module of the present invention; Figure 6 This is a schematic diagram of the dynamic compensation module of the present invention; Figure 7 This is a schematic diagram of the weld seam trimming module of the present invention; Figure 8 This is a schematic diagram of the system workflow of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1, please refer to Figure 1 and Figure 8A positioning welding robot system based on active and passive vision sensing includes a weld seam tracking module for monitoring the weld seam and welding status, and adjusting the welding torch position. The weld seam tracking module comprises a vision sensing unit, a deviation detection unit, and an automatic deviation correction unit. The vision sensing unit is connected to the deviation detection unit via Bluetooth, and the deviation detection unit is also connected to the automatic deviation correction unit via Bluetooth. The vision sensing unit scans the weld seam image using a binocular camera, generates a low-resolution point cloud, tracks the coarse positioning area, projects laser stripes into the coarse positioning area, and acquires video footage of the welding process. The image is used to generate a weld point cloud. The deviation detection unit extracts features from the video image data, including the upper and lower edge features of the welding torch and the upper and lower bevel features of the weld. Combined with the weld point cloud data, it calculates the deviation between the centerline of the welding torch and the centerline of the weld. The automatic correction unit calculates the direction and distance that the welding torch needs to be adjusted based on the deviation, and adjusts the position and posture of the welding torch to control the deviation between the welding torch and the weld within a preset range. The weld tracking module is connected to a video frame splitting module via Bluetooth. The video frame splitting module is used to split the video images during the welding process into images frame by frame for image processing.

[0023] Furthermore, in the passive vision stage, the binocular camera acquires weld seam images via arc light or captures weld seam images using natural light. The vision sensing unit acquires weld seam images from each of the two cameras in the binocular camera, extracts the weld seam images from the two images, and performs feature point matching. Feature points are pixels with prominent geometric characteristics, such as weld seam edges and corners. Matching pixels are found in the two weld seam images; these pixels correspond to the same object point in space. Using the principle of triangulation, based on the positional differences of the matching pixels and the intrinsic and extrinsic parameters of the binocular camera, the three-dimensional coordinates of each pixel are calculated. During the three-dimensional coordinate generation process: it is assumed that the optical centers of the two cameras in the binocular camera are O1 and O2, respectively. 2. The corresponding pixels P1 and P2 of feature point P in the two images are calculated based on the camera's intrinsic and extrinsic parameters. The equations of light rays O1P1 and O2P2 are calculated, and the intersection of the two light rays is solved to obtain the three-dimensional coordinates of feature point P. Using point cloud generation tools, such as PCL and VTK, the three-dimensional coordinate data is converted into a low-resolution point cloud. The visual sensing unit uses feature extraction algorithms to extract the features of the weld in the point cloud data, such as the weld edge, width, and height, thereby determining the approximate position of the weld in the point cloud data. Then, the coarse positioning area of ​​the weld is identified and tracked. The coarse positioning area is an estimate of the approximate position of the weld, thus providing a range constraint for the subsequent accurate tracking of the weld position. In the active vision stage, the laser emitter or structured light projection module integrated into the binocular camera projects laser stripes in the coarse positioning area. These laser stripes form light bands on the weld surface, reflecting the precise features of the weld. The vision sensing unit acquires video images of the welding process through the binocular camera. These images include the light spot pattern where the laser stripes intersect with the weld, background information around the weld, etc. The deformation information of the laser stripes reflects the precise position and shape of the weld. The video frame splitting module divides the video images from the vision sensing unit into individual image frames. Based on the image information split and processed by the video frame splitting module, the vision sensing unit generates weld point cloud data through the same steps as in the passive vision stage. The system reflects the three-dimensional morphological information of the weld. The vision sensing unit registers the generated low-resolution point cloud with the high-precision weld point cloud using the ICP algorithm, iterating the nearest point, adjusting the pose of the point cloud, and continuously reducing the error between the two point clouds to generate unified high-precision point cloud data. By fusing the data from active vision and passive vision, a more accurate and stable weld point cloud is generated. In addition, while the binocular camera acquires video images of the weld, the vision sensing unit uses a high-speed camera to acquire images of the weld pool, obtaining information about the welding torch and the weld pool, thereby realizing dynamic monitoring of the welding process by passive vision. Passive vision provides preliminary positioning of the weld, while active vision performs precise measurement by projecting laser stripes.

[0024] The deviation detection unit receives video image data collected by the vision sensing unit, extracts features from the image data through image processing algorithms, identifies the upper and lower edge positions of the welding torch in the image, and provides a benchmark for subsequent deviation calculation. It also extracts weld bevel features through edge detection methods to identify the weld bevel shape and size. The deviation detection unit integrates weld image features with point cloud features, generates a 3D model of the weld through a 3D reconstruction algorithm, and calculates the shortest distance from each point on the welding torch centerline to the weld centerline in real time using the 3D model. The maximum value is taken as the distance deviation. The unit also calculates the angle between the welding torch centerline and the weld centerline as the angular deviation. The automatic correction unit receives the real-time deviation calculated by the deviation detection unit and calculates the required XYZ axis translation of the welding torch. Combined with the rotation axis, it calculates the welding torch tilt angle and position. Based on the calculation results, the automatic correction unit adjusts the position and attitude of the welding torch in real time to ensure that the deviation between the welding torch and the weld is controlled within a preset range.

[0025] Example 2, please refer to Figure 2 and Figure 8A positioning welding robot system based on active and passive vision sensing is disclosed. The weld tracking module is connected to a video frame splitting module via Bluetooth. The video frame splitting module is used to split the video images during the welding process frame by frame into images for image processing. The video frame splitting module includes a frame splitting processing unit, a field-of-view correction unit, and an image enhancement unit. The frame splitting processing unit is connected to the field-of-view correction unit via Bluetooth, the field-of-view correction unit is connected to the image enhancement unit via Bluetooth, and the vision sensing unit is connected to the frame splitting processing unit via Bluetooth. The frame splitting processing unit uses a video frame splitting algorithm to split the video images during the welding process into individual image frames, and stores the image frames sequentially in a database. The field-of-view correction unit uses an image processing algorithm to identify contours in the image frames, select regions of interest, calculate the tilt angle of the image, and use a geometric transformation algorithm to rotate and correct the image to restore the horizontal and vertical states of the regions of interest. The image enhancement unit performs histogram analysis on the image frames to obtain the grayscale value range of the regions of interest, and uses an image enhancement algorithm to enhance the image frames.

[0026] Furthermore, the frame splitting processing unit uses a video frame splitting algorithm to convert the continuous video stream into image frames, and performs fine splitting of the video data generated during the welding process to ensure that each frame image is accurately extracted. In addition, the frame splitting processing unit introduces parallel processing technology and uses multi-core CPUs or GPUs for acceleration, thereby realizing fast frame splitting processing of the video stream. Using distributed storage, the image frames are stored in the database in order and numbered. The field-of-view correction unit extracts image frames from the database and analyzes each frame image output and stored by the frame splitting processing unit using image processing algorithms. This accurately identifies contour features in the image. Simultaneously, the field-of-view correction unit automatically selects regions of interest on the image. These regions typically include the upper and lower edges of the welding torch, the weld seam area, and the surrounding area of ​​the weld seam—all containing key information about the welding torch and weld seam. The field-of-view correction unit detects geometric features within these regions and performs mathematical calculations using the least squares method to obtain the image's tilt angle. Then, it uses a geometric transformation algorithm to rotate and correct the image, thereby restoring the horizontal and vertical states of the regions of interest. (Figure) The image enhancement unit performs histogram analysis on the image frames after frame splitting and correction to obtain the grayscale value range of the region of interest. Based on the distribution of grayscale values, it performs image enhancement processing to make the grayscale value distribution more uniform. In addition, the image enhancement unit combines multiple techniques such as adaptive histogram equalization and Laplacian pyramid enhancement. When there are situations such as high-brightness arc light and low contrast in the region of interest in the weld seam image, the image enhancement unit performs corresponding image enhancement processing, such as contrast enhancement and brightness adjustment, thereby improving the accuracy of features when the weld seam tracking module extracts key features such as welding guns and weld seams in the image.

[0027] Example 3, please refer to Figure 3 and Figure 8 A welding seam locating robot system based on active and passive visual sensing is disclosed. The welding seam tracking module is connected to a welding seam defect module via Bluetooth. The welding seam defect module is used for weld seam defect detection. The welding seam defect module includes a region extraction unit, a defect segmentation unit, and a defect recognition unit. The region extraction unit is connected to the defect segmentation unit via Bluetooth, and the defect segmentation unit is connected to the defect recognition unit via Bluetooth. The region extraction unit calculates the curvature value of each data point in the weld seam point cloud, initially segments the weld seam region and other regions, performs point cloud slicing processing on the other regions, defines the weld seam region, and merges them to obtain a complete weld seam region point cloud. The defect segmentation unit divides the weld seam region point cloud into a grid, compares the number of points in the grid with a preset threshold, segments the burn-through region and planar region, and uses a region growth segmentation algorithm to process the planar region and segment the defect region. The defect recognition unit extracts the point cloud features of the defect region, constructs a weld seam defect recognition and classification model, and automatically identifies and classifies defects, including weld beads, depressions, weld slag, and porosity.

[0028] Furthermore, the region extraction unit receives the weld point cloud data generated by the weld tracking module, calculates the curvature value of each data point in the weld point cloud, and evaluates the curvature value of each data point in the weld point cloud. Curvature reflects the geometric features of the weld point cloud surface; high curvature areas correspond to key parts such as weld edges, bends, or defects. The region extraction unit identifies the boundary between the weld area and the background area by setting a curvature threshold. For example, in the weld point cloud, the curvature value of the weld centerline or edge is higher, while the curvature value of the flat background area is lower. By setting the threshold, the region extraction unit can filter out the weld areas that need to be processed, thus initially separating the weld area from other areas. After obtaining the preliminary segmentation results, the region extraction... The unit employs point cloud slicing technology, using slicing software to cut the weld point cloud of other regions to obtain two-dimensional slices. The sliced ​​point cloud data undergoes detailed feature extraction, such as further edge detection, to obtain the subtle structural features of other regions, thereby further defining the range of the weld region. By merging the point cloud data, a complete weld region point cloud is obtained. Before the region extraction unit performs slicing fitting, the approximate weld region is initially segmented. Other regions outside the approximate weld region are sliced, and the weld region is further defined within these other regions. By merging the initially segmented weld region and the further defined weld region, a complete weld region is obtained while improving the processing efficiency of the region extraction unit. The defect segmentation unit employs grid partitioning technology to divide the weld area point cloud generated by the region extraction unit into multiple small grids. It calculates the number of points within each grid and compares it to a preset threshold. Points below the threshold are marked as burn-through areas, while others are marked as planar areas. For planar areas, the defect segmentation unit uses a region growing segmentation algorithm. It selects multiple seed points and expands the region based on the similarity of the point cloud data, adding points that meet the similarity criteria to the expanded region and updating the boundary point set, thereby separating the weld areas within the planar region. The defect identification unit separates the point cloud of the defect region from the surrounding points, which differs significantly in terms of normal direction, geometry, etc., and labels these extended regions as defect regions. The defect identification unit extracts features from the point cloud of the defect region separated by the defect segmentation unit, and constructs a model for identifying and classifying weld defects. Through the labeled weld defect sample data, the feature distribution and classification rules of the weld defect region are identified. The defect identification unit matches the features in the model with the features of the point cloud of the defect region to identify the defect type of the weld region, such as weld beads, depressions, weld slag, and porosity, and classifies them according to the type of weld defect.

[0029] Example 4, please refer to Figure 1 , Figure 2 , Figure 4 and Figure 8 A positioning welding robot system based on active and passive vision sensing is disclosed. The weld seam tracking module is connected to a sensor fusion module via Bluetooth. The sensor fusion module integrates data from different sensors and adjusts the sensor parameters and operating modes. The sensor fusion module includes a multi-dimensional fusion unit, a blind zone compensation unit, and a dynamic coordination unit. The multi-dimensional fusion unit is connected to the blind zone compensation unit via Bluetooth, and the dynamic coordination unit is also connected to the dynamic coordination unit via Bluetooth. The multi-dimensional fusion unit collects real-time monitoring data from the sensor network, including force sensors, arc sensors, and temperature sensors, and combines it with image data from the welding process, using a data fusion algorithm to fuse the multi-dimensional data. The blind zone compensation unit performs pattern recognition based on the multi-dimensional data, identifies sensor blind zones, determines missing multi-dimensional data, and uses the multi-dimensional data to compensate for blind zone information from other sensors. The dynamic coordination unit analyzes the multi-dimensional data in real time, identifies changes in the weld seam's position, shape, and welding environment, and dynamically adjusts the sensor network's operating parameters and welding parameters according to the specific requirements of the welding task. The system includes a weld seam tracking module for monitoring the weld seam and welding status, and adjusting the welding torch position. The weld seam tracking module comprises a vision sensing unit, a deviation detection unit, and an automatic deviation correction unit. The vision sensing unit is connected to the deviation detection unit via Bluetooth, and the deviation detection unit is connected to the automatic deviation correction unit via Bluetooth. The weld seam tracking module is also connected to a video frame splitting module via Bluetooth. This video frame splitting module is used to split video images during the welding process into individual images for image processing. The video frame splitting module comprises a frame splitting processing unit, a field-of-view correction unit, and an image enhancement unit. The frame splitting processing unit is connected to the field-of-view correction unit via Bluetooth, the field-of-view correction unit is connected to the image enhancement unit via Bluetooth, and the vision sensing unit is connected to the frame splitting processing unit via Bluetooth.

[0030] Furthermore, the multi-dimensional fusion unit collects monitoring data from multiple sensors, such as force sensors, arc sensors, and temperature sensors. Through data fusion algorithms, such as Kalman filtering, particle filtering, and neural networks, it fuses multi-dimensional data, performs spatiotemporal alignment, weight allocation, and optimized combination of data from different sensors, and integrates real-time monitoring data from multiple sensors to provide the system with comprehensive perception of the welding process. For example, force sensor data reflects the contact force between the welding torch and the weld, arc sensor data reflects the stability of the arc and the welding quality, and temperature sensor data reflects the heat distribution in the welding area. In addition, the multi-dimensional fusion unit supports the fusion of multi-source heterogeneous data, combining active visual data and passive visual data from the visual sensing unit. The blind zone compensation unit analyzes the multidimensional data fused by the multidimensional fusion unit to identify data gaps caused by the complexity of the welding environment and the limitations of the sensors themselves. It also identifies data gaps or anomalies caused by certain areas or angles in the welding process that cannot be fully monitored by a single sensor. Based on these gaps or anomalies, the blind zone compensation unit identifies the blind zone information, calls relevant data from other sensors, and completes the blind zone information through data interpolation and fusion. For example, when smoke or dust obscures the video image in the visual sensing unit, the blind zone compensation unit uses the arc current waveform from the arc sensor data and the temperature field distribution from the temperature sensor to generate a point cloud to complete the data. Furthermore, the blind zone compensation unit can dynamically adjust the weight parameters of each sensor through fuzzy logic. The dynamic coordination unit analyzes the multidimensional data processed by the blind zone compensation unit in real time, identifies the position and shape of the weld, and changes in the welding environment. Based on the specific needs of the welding task, it dynamically adjusts welding parameters, such as adjusting the welding torch oscillation amplitude, welding speed, and current, and adjusts the working parameters of the sensor network, such as setting the corresponding sensor to the dominant mode based on the weight parameters of each sensor, and shutting down redundant sensors in stable welding sections. In addition, the visual sensing unit in the weld seam tracking module dynamically adjusts the overall video acquisition frame rate of the binocular camera based on the welding parameters adjusted by the dynamic coordination unit, avoiding video blurring during high-speed welding. This ensures that the subsequent video frame splitting module's frame splitting processing unit obtains clear images. At the same time, the frame splitting processing unit, in conjunction with the sensor fusion module, adds real-time welding parameters and environmental data to each split frame image, generating spatiotemporally aligned multidimensional data to provide multidimensional data for subsequent weld seam defects.

[0031] Example 5, please refer to Figure 5 , Figure 7 and Figure 8 A positioning welding robot system based on active and passive vision sensing is disclosed. The weld defect module is connected to a weld trimming module via Bluetooth. The weld trimming module is used to handle weld defects. The weld trimming module includes a trimming planning unit and a trajectory planning unit. The trimming planning unit is connected to the trajectory planning unit via Bluetooth, and the trimming planning unit is also connected to a defect recognition unit via Bluetooth. The trimming planning unit collects the results of weld defect recognition and classification, including defect type, defect location, and defect size information. Combining the physical properties of the welding material and welding process parameters, it formulates a weld trimming strategy, including the selection of trimming tools, the planning of the machining swing angle, and the planning of the feed speed. The trajectory planning unit calculates and optimizes the collision-free machining path of the trimming tool based on the weld trimming strategy and weld defect data using a path planning algorithm. The weld seam tracking module is connected to a weld seam defect module via Bluetooth. The weld seam defect module is used for weld seam defect detection. The weld seam defect module includes a region extraction unit, a defect segmentation unit, and a defect recognition unit. The region extraction unit is connected to the defect segmentation unit via Bluetooth, and the defect segmentation unit is connected to the defect recognition unit via Bluetooth. The weld seam tracking module is connected to a multi-robot collaboration module via Bluetooth. The multi-robot collaboration module is used to coordinate the path planning and action timing of multiple robots. The multi-robot collaboration module includes a spatial collaboration unit and a task allocation unit. The spatial collaboration unit is connected to the task allocation unit via Bluetooth, and the visual sensing unit is connected to both the spatial collaboration unit and the task allocation unit via Bluetooth. The spatial collaboration unit constructs a cooperative kinematics model, calculates the optimal working range of each robot, and coordinates the action timing of multiple robots. The task allocation unit uses a task decomposition and planning algorithm to allocate welding tasks and dynamically adjusts the task allocation scheme according to the priority of the welding tasks and the capabilities of the robots.

[0032] Furthermore, the trimming planning unit receives the weld defect identification and classification results analyzed by the weld defect module, including defect type, defect location, and defect size information. It constructs a welding material database based on the physical properties of the welding material. Combining welding process parameters and weld defect information, the trimming planning unit formulates a weld trimming strategy, such as what trimming tool is needed for the corresponding defect type. It predicts the surface roughness of the weld after trimming based on the robot learning model, simulates different processing swing angles and feed speeds, and selects the optimal parameters. The trajectory planning unit constructs a kinematic model of the trimming tool based on the weld trimming strategy formulated by the trimming planning unit and the weld defect data from the weld defect module. It uses the ant colony algorithm to solve for the optimal trajectory and generates a smooth trimming path through the curvature continuous interpolation algorithm, thus completing the optimization of the processing path. During the process of repairing weld defects by controlling the trimming robot according to the processing path generated by the trajectory planning unit, the spatial coordination unit couples the models of the welding robot and the trimming robot to construct a cooperative kinematics model. Based on the predicted action sequence and spatial interference of multiple robots in the model, it calculates the optimal working range of each robot, coordinates the action sequence of multiple robots, and controls multiple robots to work collaboratively while avoiding robot collisions, so as to realize that multiple robots can perform welding tasks or welding and trimming tasks simultaneously. The task allocation unit uses task decomposition and planning algorithms to decompose a whole welding task into multiple sub-tasks, constructs a task priority model, allocates the priority of sub-tasks, and dynamically adjusts the task allocation scheme according to the priority of welding tasks and the capabilities of robots.

[0033] Example 6, please refer to Figure 6 and Figure 8 A positioning welding robot system based on active and passive vision sensing is disclosed. The image enhancement unit is connected to a dynamic compensation module via Bluetooth. The dynamic compensation module monitors the light intensity of the welding area in real time and adjusts the parameters of the binocular camera. The dynamic compensation module includes a light monitoring unit, a camera adjustment unit, and a quality assessment unit. The light monitoring unit is connected to the camera adjustment unit via Bluetooth, and the camera adjustment unit is connected to the quality assessment unit via Bluetooth. The image enhancement unit is also connected to the quality assessment unit via Bluetooth. The light monitoring unit monitors the light intensity of the welding area in real time using a light intensity sensor mounted on the binocular camera, analyzes the trend of light intensity changes, and identifies fluctuations in light intensity. The camera adjustment unit sets a threshold range for light intensity and dynamically adjusts the parameters of the binocular camera, including exposure time, gain setting, and white balance parameters, based on changes in light intensity and image quality assessment results. The quality assessment unit performs quality assessment on the enhanced image, sets a feedback mechanism for image quality that does not meet the standard, and records the camera parameters and corresponding light intensity data after each adjustment.

[0034] Furthermore, during the welding process, the light intensity in the welding area is affected by factors such as the arc light, changes in ambient light and shadow, and laser stripes projected onto the weld area. The light monitoring unit collects real-time light intensity monitoring data from the light intensity sensor. By collecting and analyzing continuous monitoring data, the light monitoring unit obtains the trend of light intensity changes. The light monitoring unit receives image data processed by the image enhancement unit and, combined with the image data and the light intensity at corresponding time points, identifies light fluctuations that negatively impact the welding image quality. The camera adjustment unit receives the light fluctuation data affecting image quality analyzed by the light monitoring unit and, combined with historical light intensity data and corresponding image quality data, sets the light intensity level accordingly. The camera adjustment unit sets a threshold range for illumination intensity. When the camera adjustment unit detects that the illumination intensity exceeds or falls below these thresholds, it dynamically adjusts the parameters of the binocular camera based on the changes in illumination intensity. After triggering the feedback mechanism, the camera adjustment unit fine-tunes the parameters of the binocular camera based on the image quality assessment results until the optimal image effect is achieved. The quality assessment unit performs a quality assessment on the image processed by the image enhancement unit and sets a feedback mechanism for image quality that does not meet the standard. When the image quality does not meet the standard, the quality assessment unit notifies the camera adjustment unit to adjust the parameters of the binocular camera. At the same time, the quality assessment unit records the camera parameters and corresponding illumination intensity data after each adjustment, providing a data basis for subsequent welding quality analysis.

[0035] Working principle: During the welding process, the weld seam tracking module collects video of the welding process in real time. The video frame splitting module splits the video stream from the weld seam tracking module into individual image frames and performs correction and enhancement processing on the images. Based on the images processed by the video frame splitting module, the weld seam tracking module generates weld seam point cloud data. The sensor fusion module maps the multidimensional data collected by the sensor network to the image split by the video frame splitting module. The sensor fusion module integrates the multidimensional data and image data, compensates for blind spot information, generates point cloud data, and combines the point cloud data and weld point cloud data to obtain complete point cloud data. The weld tracking module calculates the deviation between the welding torch centerline and the weld centerline based on the point cloud data, and adjusts the position and orientation of the welding torch. At the same time, the weld defect module calculates the curvature value of the point cloud data to identify weld defects.

[0036] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A positioning welding robot system based on active and passive vision sensing, characterized in that: It includes a weld seam tracking module, which is used to monitor the weld seam and welding status and adjust the position of the welding torch; The weld seam tracking module includes: a vision sensing unit, a deviation detection unit, and an automatic correction unit. The vision sensing unit is connected to the deviation detection unit via Bluetooth, and the deviation detection unit is connected to the automatic correction unit via Bluetooth. The visual sensing unit scans the weld seam image with a binocular camera to generate a low-resolution point cloud, tracks the coarse positioning area, projects laser stripes in the coarse positioning area, and collects video images of the welding process to generate the weld seam point cloud. The deviation detection unit extracts features from the video image data, extracts the upper and lower edge features of the welding torch and the upper and lower bevel features of the weld, and calculates the deviation between the center line of the welding torch and the center line of the weld by combining the weld point cloud data. The automatic correction unit calculates the direction and distance that the welding torch needs to be adjusted based on the deviation, adjusts the position and posture of the welding torch, and controls the deviation between the welding torch and the weld within a preset range. The weld seam tracking module is connected to a video frame splitting module via Bluetooth. The video frame splitting module is used to split the video images during the welding process into images frame by frame for image processing. The video frame splitting module includes: a frame splitting processing unit, a field of view correction unit, and an image enhancement unit. The frame splitting processing unit is connected to the field of view correction unit via Bluetooth, the field of view correction unit is connected to the image enhancement unit via Bluetooth, and the visual sensing unit is connected to the frame splitting processing unit via Bluetooth. The frame splitting processing unit uses a video frame splitting algorithm to split the video images during the welding process into individual image frames, and stores the image frames in the database in sequence. The field correction unit uses image processing algorithms to identify contours in image frames, select regions of interest, calculate the tilt angle of the image, and use geometric transformation algorithms to rotate and correct the image to restore the horizontal and vertical states of the regions of interest. The image enhancement unit performs histogram analysis on the image frame to obtain the gray value range of the region of interest, and then uses an image enhancement algorithm to enhance the image frame. The weld seam tracking module is connected to a sensor fusion module via Bluetooth. The sensor fusion module is used to integrate data from different sensors and adjust the parameters and working modes of the sensors. The sensor fusion module includes: a multi-dimensional fusion unit, a blind spot compensation unit, and a dynamic coordination unit. The multi-dimensional fusion unit is connected to the blind spot compensation unit via Bluetooth, and the multi-dimensional fusion unit is connected to the dynamic coordination unit via Bluetooth. The multi-dimensional fusion unit collects real-time monitoring data from the sensor network, including force sensors, arc sensors, and temperature sensors. Combined with image data from the welding process, the multi-dimensional data is fused using a data fusion algorithm. The blind spot compensation unit performs pattern recognition based on multidimensional data to identify the blind spots of the sensors, determine the missing multidimensional data, and use the multidimensional data to compensate for the blind spot information of other sensors. The dynamic collaborative unit analyzes multi-dimensional data in real time, identifies the location, shape, and changes in the welding environment of the weld, and dynamically adjusts the working parameters of the sensor network and the welding parameters according to the specific requirements of the welding task.

2. The positioning welding robot system based on active and passive vision sensing according to claim 1, characterized in that: The weld seam tracking module is connected to a weld seam defect module via Bluetooth, and the weld seam defect module is used for weld seam defect detection. The weld defect module includes: a region extraction unit, a defect segmentation unit, and a defect recognition unit. The region extraction unit is connected to the defect segmentation unit via Bluetooth, and the defect segmentation unit is connected to the defect recognition unit via Bluetooth. The region extraction unit calculates the curvature value of each data point in the weld point cloud, initially segments the weld region and other regions, performs point cloud slicing on the other regions, defines the weld region, and merges them to obtain the complete weld region point cloud. The defect segmentation unit divides the point cloud of the weld area into a grid, compares the number of points in the grid with a preset threshold, and segments the burn-through area and the planar area. The planar area is processed by the region growth segmentation algorithm, and the defect area is segmented. The defect identification unit extracts the point cloud features of the defect area, constructs a model for identifying and classifying weld defects, and automatically identifies and classifies defects, including weld beads, depressions, weld slag, and porosity.

3. The positioning welding robot system based on active and passive vision sensing according to claim 2, characterized in that: The weld defect module is connected to a weld repair module via Bluetooth, and the weld repair module is used to process weld defects. The weld seam trimming module includes: a trimming planning unit and a trajectory planning unit. The trimming planning unit is connected to the trajectory planning unit via Bluetooth, and the trimming planning unit is connected to the defect recognition unit via Bluetooth. The finishing planning unit collects the results of weld defect identification and classification, including defect type, defect location, and defect size information. Combined with the physical properties of the welding material and welding process parameters, it formulates weld finishing strategies, including the selection of finishing tools, the planning of machining angle, and the planning of feed speed. Based on the weld repair strategy and weld defect data, the trajectory planning unit uses a path planning algorithm to calculate and optimize the collision-free processing path of the repair tool.

4. The positioning welding robot system based on active and passive vision sensing according to claim 1, characterized in that: The image enhancement unit is connected to a dynamic compensation module via Bluetooth. The dynamic compensation module is used to monitor the light intensity of the welding area in real time and adjust the parameters of the binocular camera. The dynamic compensation module includes a light monitoring unit, a camera adjustment unit, and a quality assessment unit. The light monitoring unit is connected to the camera adjustment unit via Bluetooth, the camera adjustment unit is connected to the quality assessment unit via Bluetooth, and the image enhancement unit is connected to the quality assessment unit via Bluetooth. The illumination monitoring unit monitors the illumination intensity of the welding area in real time through the illumination intensity sensor mounted on the binocular camera, analyzes the trend of illumination intensity changes, and identifies the fluctuations in illumination intensity. The camera adjustment unit sets a threshold range for light intensity and dynamically adjusts the parameters of the binocular camera, including exposure time, gain settings, and white balance parameters, based on changes in light intensity and image quality assessment results. The quality assessment unit evaluates the quality of the enhanced image, sets up a feedback mechanism for images whose quality does not meet the standard, and records the camera parameters and corresponding illumination intensity data after each adjustment.

5. The positioning welding robot system based on active and passive vision sensing according to claim 1, characterized in that: The weld seam tracking module is connected to a multi-robot collaboration module via Bluetooth. The multi-robot collaboration module is used to coordinate the path planning and action timing of multiple robots. The multi-machine collaboration module includes: a spatial collaboration unit and a task allocation unit. The spatial collaboration unit is connected to the task allocation unit via Bluetooth, and the visual sensing unit is connected to both the spatial collaboration unit and the task allocation unit via Bluetooth. The spatial collaborative unit constructs a collaborative kinematics model, calculates the optimal working range of each robot, and coordinates the action sequence of multiple robots. The task allocation unit uses task decomposition and planning algorithms to allocate welding tasks and dynamically adjusts the task allocation scheme based on the priority of the welding tasks and the capabilities of the robot.

6. The positioning welding robot system based on active and passive vision sensing according to claim 1, characterized in that: The generation of the low-resolution point cloud includes the following steps: Image matching: Feature point matching is performed on two weld seam images scanned by a binocular camera to match the corresponding pixels; Parallax calculation: Based on the positional differences of the matching points and the intrinsic and extrinsic parameters of the binocular camera, the three-dimensional coordinates of each pixel are calculated to generate point cloud data containing depth information.

7. The positioning welding robot system based on active and passive vision sensing according to claim 1, characterized in that: The generation of the weld point cloud includes the following steps: Feature extraction is performed on image frames containing the spot pattern of laser stripes intersecting with the weld and background information around the weld to extract the spot pattern of the laser stripes; Based on the shape and position information of the light spot pattern and the camera's internal and external parameters, the precise three-dimensional coordinates of the weld are calculated, and weld point cloud data is generated.

8. The positioning welding robot system based on active and passive vision sensing according to claim 1, characterized in that: The areas of interest include: the upper and lower edges of the welding torch, the weld seam area, and the area surrounding the weld seam.

Images