Machine vision based friction stir welding seam quality monitoring system and method

Abstract

The application provides a machine vision-based friction stir welding seam quality monitoring system and method, and relates to the technical field of friction stir welding. The friction stir welding seam quality monitoring system is installed in a main body of a friction stir welding device. The system comprises, which are connected in sequence, a vision module, a three-dimensional reconstruction module, a quality prediction module, a decision module and a data transmission module. The vision module, the three-dimensional reconstruction module, the quality prediction module, the decision module and the data transmission module are electrically connected. The welding seam quality monitoring method realizes three-dimensional reconstruction of the surface state of a welding seam in a welding process through the friction stir welding seam quality monitoring system, and utilizes a convolutional neural network to perform online nondestructive detection and evaluation on the surface defects, internal defects and welding quality of the friction stir welding seam, thereby filling the technical gap in online intelligent detection of internal defects of friction stir welding.

Description

Technical Field

[0001] This invention relates to the field of friction stir welding technology, and in particular to a machine vision-based system and method for monitoring the quality of friction stir welds. Background Technology

[0002] Friction stir welding (FSW) is a green and efficient solid-state joining technology that has been widely used in aerospace, shipbuilding, automotive, and railway industries. However, the FSW process is often accompanied by many defects, such as flash, weld thinning, grooves, and voids, which are not conducive to obtaining high-quality joints.

[0003] In recent years, with the development of friction stir welding technology, the weld quality inspection technology for friction stir welding has also developed rapidly. Existing inspection technologies mostly focus on monitoring the welding process and detecting post-weld defects. For example, one method uses vision and displacement sensors to acquire and display images of the welding process in real time. The displacement sensor precisely controls the shoulder pressure to obtain workpieces with consistent welding depth, while the vision sensor provides a visual image display. However, this method cannot inspect the quality of the completed weld. Another example is using machine vision to detect surface defects in the weld. By convolving the acquired weld images, surface defects are identified. However, this method does not provide adjustment suggestions or defect handling solutions for the identified defects. In addition to the above methods, a machine vision-based aluminum alloy friction stir welding control system and method have been proposed. This method can not only output the defect type and corresponding quantitative parameters, but also constructs a three-dimensional surface without welding defects using orthogonal experimental design, and optimizes the welding parameters through difference fitting. However, this method is also limited to the identification of surface defects in the weld. Currently, research on the detection of weld quality in friction stir welding is either limited to the identification of surface defects or only the detection of internal defects after welding is completed, without realizing the prediction of internal defects by extracting surface features during the welding process. Summary of the Invention

[0004] To address the shortcomings of the existing technologies, this invention proposes a machine vision-based system and method for monitoring the quality of friction stir welding seams. This system tracks the weld seam in real time during the friction stir welding process and quickly obtains surface information of the welded area. A visual 3D model is constructed using 3D reconstruction technology. During the welding process, it enables the prediction of internal defects in the weld seam and the quality of the weld joint, the marking of defect locations, and the implementation of a process parameter decision feedback scheme based on the quality of the weld joint.

[0005] The first aspect of this invention proposes a machine vision-based friction stir welding weld quality monitoring system, which is installed in a friction stir welding equipment. The friction stir welding equipment includes a spindle, a stirring head, a workpiece, a worktable, and a friction stir welding CNC system. The system includes a vision module, a 3D reconstruction module, a quality prediction module, a decision module, and a data transmission module that are electrically connected in sequence. The data transmission module is electrically connected to the friction stir welding CNC system.

[0006] The vision module is used to acquire the laser stripe pattern projected onto the weld surface and send the acquired laser stripe pattern to the three-dimensional reconstruction module.

[0007] The three-dimensional reconstruction module is used to draw a weld point cloud model and a weld three-dimensional model based on the laser stripe pattern acquired by the vision module.

[0008] The quality prediction module is used to evaluate weld quality. It inputs process parameters including spindle speed, welding speed, and shoulder pressing amount, as well as the point cloud model obtained from the three-dimensional reconstruction module, into a pre-trained convolutional neural network to evaluate the welding quality of the completed welding parts and output welding quality prediction results. The welding quality prediction results output by the convolutional neural network include the tensile strength of the weld joint, defect type, and evaluation index corresponding to the defect.

[0009] The decision module is used to determine whether the welding quality prediction result meets the welding requirements. If yes, the current process parameters are maintained and welding continues; if no, a real-time adjustment plan is generated to adjust the process parameters.

[0010] The data transmission module is used to access the friction stir welding CNC system to read and write data. By accessing the relevant data of each axis in the friction stir welding CNC system, it obtains process parameters. By calling the library functions of the dynamic link library encapsulated in the CNC equipment of the friction stir welding CNC system, it performs read and write operations on process parameters and controls the welding process.

[0011] The vision module further includes:

[0012] A camera module for capturing images containing laser stripe patterns on the weld surface;

[0013] Furthermore, the camera module consists of a first array camera, a line laser projector, and a second array camera, which are fixed on the main shaft of the friction stir welding equipment according to the binocular vision model and the triangulation principle. The light plane of the line laser projector is coplanar with the baseline in the binocular vision model.

[0014] The 3D reconstruction module further comprises three modules in sequence, namely:

[0015] The image preprocessing module is used to analyze the characteristics of the images acquired by the first array camera and the second array camera, filter the noise in the images, and obtain the three-dimensional geometric information of the center line of the laser stripe pattern.

[0016] The point cloud processing module is used to unify the three-dimensional geometric information of the center line of the laser stripe pattern into the world coordinate system; acquire the point cloud data of the weld and complete the point cloud data to obtain a complete weld point cloud model; segment the weld point cloud model to obtain the point cloud model of the flash area and the point cloud model of the weld thinning area.

[0017] The calculation module is used to calculate the volume of the flash region point cloud model and the weld thinning region point cloud model based on the flash region point cloud model and the weld thinning region point cloud model obtained from the point cloud processing module, and to calculate the volume of the flash region point cloud model and the weld thinning region point cloud model using the integration method.

[0018] The image preprocessing module further includes:

[0019] The image filtering and denoising unit is used to denoise the image and eliminate interference information;

[0020] The image enhancement unit is used to enhance the image and highlight the details in the image;

[0021] The threshold segmentation unit is used to extract the region of interest (ROI) in the image and separate the laser stripe pattern on the weld surface from the background.

[0022] The light stripe centerline extraction unit is used to extract the laser stripe pattern on the weld surface and obtain two light stripe centerlines with sub-pixel precision of the laser stripe pattern.

[0023] The binocular stereo matching unit is used to calculate the positional deviation between corresponding points in the image and obtain the three-dimensional geometric information of the center line of the laser stripe pattern;

[0024] The point cloud processing module further includes:

[0025] The point cloud stitching unit is used to translate and rotate the acquired three-dimensional geometric information of the light stripe centerline to the world coordinate system to construct a weld point cloud model.

[0026] The point cloud completion unit is used to repair the missing point cloud in the weld point cloud model. It estimates the complete point cloud from the missing point cloud to obtain the complete weld point cloud model.

[0027] The point cloud segmentation unit is used to segment the flash area point cloud model and the weld thinning area point cloud model from the complete weld point cloud model.

[0028] A second aspect of this invention proposes a machine vision-based method for monitoring the quality of friction stir welds, the method comprising:

[0029] Step 1: The user sets the process parameters for friction stir welding according to the welding task, and performs stirring pin interference inspection on the friction stir welding equipment, the workpiece to be welded, the first array camera, the line laser projector and the second array camera to establish a reference plane.

[0030] The first array camera, the line laser projector, and the second array camera are fixed on the main shaft of the friction stir welding equipment according to the binocular vision model and the triangulation principle, wherein the light plane of the line laser projector is coplanar with the baseline in the binocular vision model.

[0031] Step 2: The friction stir welding equipment starts welding according to the set process parameters and acquires an image containing the laser stripe pattern on the weld surface;

[0032] Step 3: Perform image preprocessing on the acquired image containing the laser stripe pattern on the weld surface, and extract the two sub-pixel precision light stripe center lines from the laser stripe pattern to obtain the three-dimensional point cloud data of the weld in the current state.

[0033] Step 4: Construct a weld point cloud model based on the current state of the weld 3D point cloud data, segment the weld point cloud model to obtain the flash area point cloud model and the weld thinning area point cloud model, and calculate the volume U1 of the flash area point cloud model and the volume U2 of the weld thinning area point cloud model based on the reference plane.

[0034] Step 5: Obtain the actual process parameters of friction stir welding at the current moment, and construct the actual process parameter matrix L. t ;

[0035] Step 6: Convert the actual process parameter matrix L t By integrating the weld point cloud model, a three-dimensional weld model containing real process parameters is obtained;

[0036] Step 7: Use a pre-trained convolutional neural network to extract features from the weld 3D model with fused process parameters to obtain the joint tensile strength U3 of the weld and the evaluation index corresponding to the weld defect type. Then, combine the volume U1 of the flash area point cloud model and the volume U2 of the weld thinning area point cloud model to obtain the parameter matrix U characterizing the weld joint quality.

[0037] Step 8: Determine whether the weld meets the welding requirements based on the parameter matrix U, which characterizes the weld joint quality. If it meets the welding requirements, continue welding while maintaining the current actual process parameters; otherwise, adjust U and L. t Input a pre-trained backpropagation neural network to establish a real-time adjustment scheme and optimize the current real process parameters;

[0038] Step 9: Control the friction stir welding equipment to continue welding according to the optimized actual process parameters of friction stir welding, and repeat steps 2-8 until the welding is completed;

[0039] Furthermore, the image preprocessing described in step 4 includes: image filtering and denoising, image enhancement, and threshold segmentation.

[0040] The beneficial effects of adopting the above technical solution are as follows:

[0041] (1) The method of the present invention analyzes the surface features of the weld based on machine vision, realizes the three-dimensional reconstruction of the surface state of the weld during the welding process, and uses convolutional neural network to realize online non-destructive detection and evaluation of surface defects, internal defects and welding quality of friction stir weld. The internal defects of the joint can be intuitively displayed in the three-dimensional model. The method of the present invention fills the technical gap of online intelligent detection of internal defects in friction stir welding.

[0042] (2) The system of this invention performs online quality prediction of the weld seam through a quality prediction module, and uses a decision module to smooth and compensate for the pressing deviation caused by factors such as the processing error of the structure to be welded and changes in heat dissipation during the welding process. This reduces the quality fluctuation caused by changes in the pressing amount of the shoulder during the welding process, thereby effectively reducing the generation of defects and improving the welding quality. In addition, while performing online welding quality prediction, the system of this invention marks the location of weld defects and proposes a repair welding plan, thereby guiding subsequent joint repair welding operations, effectively improving work efficiency and product manufacturing yield, ensuring the process stability of the weld seam, and improving the joint quality. Attached Figure Description

[0043] Figure 1 This is a structural diagram of the main body of the friction stir welding equipment in this embodiment;

[0044] Figure 2 This is a structural diagram of the machine vision-based friction stir welding weld quality monitoring system of this embodiment;

[0045] Figure 3 This is a schematic diagram illustrating the working principle of the machine vision-based friction stir welding quality monitoring system in this embodiment.

[0046] Figure 4 This is a flowchart of the machine vision-based method for monitoring the quality of friction stir welding seams in this embodiment.

[0047] Figure 5 This is a schematic diagram of the real-time adjustment scheme generated by the decision model in this embodiment;

[0048] In the diagram: 1-First array camera, 2-Laser, 3-Second array camera, 4-Spindle, 5-Stirring head, 6-Workpiece, 7-Worktable, 8-Friction stir welding CNC system. Detailed Implementation

[0049] To facilitate understanding of this application, specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments. The following embodiments are illustrative of the invention but are not intended to limit its scope. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0050] The machine vision-based weld quality monitoring system for friction stir welding in this embodiment is installed in the main body of the friction stir welding equipment. For example... Figure 1 As shown, the main body of the friction stir welding equipment includes a spindle 4, a stirring head 5, a workpiece 6, a worktable 7, and a friction stir welding CNC system 8. Figure 2 As shown, the machine vision-based friction stir welding weld quality monitoring system includes, in sequence, a vision module 100, a 3D reconstruction module 200, a quality prediction module 300, a decision module 400, and a data transmission module 500. The vision module 100, the 3D reconstruction module 200, the quality prediction module 300, the decision module 400, and the data transmission module 500 are all electrically connected. The data transmission module 500 is electrically connected to the friction stir welding CNC system 8.

[0051] The vision module 100 is used to acquire the laser stripe pattern projected on the weld surface and send the acquired laser stripe pattern to the three-dimensional reconstruction module 200.

[0052] The vision module 100 further includes:

[0053] Camera module 110 is used to acquire images containing laser stripe patterns on the weld surface.

[0054] Image caching module 120 is used to store images captured by camera module 110.

[0055] The image cache control module 130 is used to determine whether the image acquired by the camera module 110 needs to be stored in the image cache module 120. When there is a conflict between image acquisition and image preprocessing, the image acquired by the camera module 110 at the current moment is placed in the image cache module 120. After the image preprocessing is completed, the image stored in the image cache module 120 is called and sent to the 3D reconstruction module 200. The output of the camera module 110 and the output of the image cache module 120 are both connected to the image cache control module 130. The output of the image cache control module 130 is divided into two paths, one of which is input to the image cache module 120 and the other is used as the output of the vision module 100.

[0056] The camera module 110 is fixed on the main shaft 4 of the friction stir welding main body equipment according to the binocular vision model and the principle of triangulation, consisting of a first array camera 1, a line laser projector 2, and a second array camera 3. The light plane of the line laser projector 2 is coplanar with the baseline in the binocular vision model.

[0057] In this embodiment, the laser stripe pattern on the weld surface acquired by the vision module 100 is determined by the image cache control module 130 to determine whether it needs to be stored in the image cache module 120, thus resolving the conflict between inconsistent data access times for image acquisition and image processing.

[0058] The three-dimensional reconstruction module 200 is used to draw a weld seam point cloud model based on the laser stripe pattern acquired by the vision module 100.

[0059] The three-dimensional reconstruction module 200 further includes three modules in sequence, namely:

[0060] The image preprocessing module 210 is used to analyze the characteristics of the images acquired by the first array camera 1 and the second array camera 3, solve the noise interference of the images, improve the image quality and image processing efficiency, and obtain the three-dimensional geometric information of the center line of the laser stripe pattern.

[0061] The point cloud processing module 220 is used to unify the three-dimensional geometric information of the center line of the laser stripe pattern into the world coordinate system, obtain the point cloud data of the weld, and complete the point cloud data to obtain a complete weld point cloud model. The weld point cloud model is segmented using feature recognition methods to obtain the point cloud model of the flash area and the point cloud model of the weld thinning area.

[0062] The calculation module 230 is used to calculate the volume of the flash region point cloud model and the weld thinning region point cloud model based on the flash region point cloud model and the weld thinning region point cloud model obtained from the point cloud processing module, and to calculate the volume of the flash region point cloud model and the weld thinning region point cloud model using the integration method.

[0063] The image preprocessing module 210 further includes:

[0064] The image filtering and denoising unit 211 is used to denoise the image and eliminate interference information.

[0065] Image enhancement unit 212 is used to enhance the image, highlight the details in the image, and improve the image quality.

[0066] The threshold segmentation unit 213 is used to extract the region of interest (ROI) in the image and separate the laser stripe pattern on the weld surface from the background. In this embodiment, since the acquired image contains both the laser stripe pattern on the weld surface and the background area, the image needs to be segmented to separate the laser stripe pattern on the weld surface from the background in order to reduce the computational load and improve processing efficiency.

[0067] The light stripe centerline extraction unit 214 is used to extract the laser stripe pattern on the weld surface and obtain two light stripe centerlines with sub-pixel precision of the laser stripe pattern.

[0068] The binocular stereo matching unit 215 is used to calculate the positional deviation between corresponding points in the image and obtain the three-dimensional geometric information of the center line of the laser stripe pattern.

[0069] The point cloud processing module 220 further includes:

[0070] Point cloud stitching unit 221 is used to translate and rotate the acquired three-dimensional geometric information of the light stripe centerline to the world coordinate system to construct a weld seam point cloud model.

[0071] Point cloud completion unit 222 is used to repair missing point clouds in the weld point cloud model. It estimates the complete point cloud from the missing point cloud to obtain the complete weld point cloud model.

[0072] Point cloud segmentation unit 223 is used to segment the flash region point cloud model and the weld thinning region point cloud model from the complete weld point cloud model.

[0073] The calculation module 230 further includes:

[0074] The flash volume calculation unit 231 is used to calculate the volume of the flash region point cloud model based on the flash region point cloud model obtained from the point cloud processing module and by using the integration method.

[0075] The weld thinning volume calculation unit 232 is used to calculate the volume of the weld thinning region point cloud model based on the weld thinning region point cloud model obtained from the point cloud processing module and by using the integration method.

[0076] In this embodiment, such as Figure 2 As shown, when the image acquired by the vision module 100 is transmitted to the three-dimensional reconstruction module 200, the acquired image is preprocessed, that is, the image filtering and denoising unit 211, the image enhancement unit 212, the threshold segmentation unit 213 and the light stripe center line extraction unit 214 are used in sequence to obtain two light stripe center lines with sub-pixel precision, and the stereo matching unit 215 is used to perform stereo matching calculation on the two light stripe center lines with sub-pixel precision to calculate the positional deviation between corresponding points in the image and obtain the three-dimensional geometric information of the light stripe center lines. Thus, the complete weld seam point cloud model is obtained through the point cloud processing module 220. In the weld seam point cloud model, the flash area point cloud model and the weld seam thinning area point cloud model are segmented. Then, the volume of the flash area point cloud model and the volume of the weld seam thinning area point cloud model in the current state are calculated by the integration method. The point cloud processing module 220's point cloud stitching unit 221 and point cloud completion unit 222, based on the motion characteristics of the gantry welding equipment, align the three-dimensional geometric information of the light stripe centerline obtained in the current state from the image preprocessing module 210 to the world coordinate system to obtain a weld point cloud model. Then, it completes the point cloud data missing in the measurement dead zone and shadow area to obtain a complete weld point cloud model. The point cloud segmentation unit 223 segments the weld point cloud model into a flash area point cloud model and a weld thinning area point cloud model based on the spatial, geometric, and texture features of the weld. The calculation module 230 calculates the volume of the flash area point cloud model and the weld thinning area point cloud model respectively using an integration method based on the segmented flash area point cloud model and weld thinning area point cloud model.

[0077] In this embodiment, the friction stir welding equipment and its motion characteristics are not limited to welding gantry and translational motion, but can be various forms of displacement platforms and motion modes. For example, when the displacement platform controls the vision module to perform translational motion, the point cloud of each motion state can be stitched together using the translation direction, translation speed, and acquisition time interval of each state to obtain a complete weld point cloud model; when the displacement platform performs rotational motion, the point cloud of each motion state can be stitched together using the rotation axis, rotation direction, rotation speed, and acquisition time interval of the rotation platform to obtain a complete weld point cloud model; when the displacement platform moves according to a pre-programmed path, the point clouds of each state are stitched together in the world coordinate system according to the known motion path to obtain a complete weld point cloud model.

[0078] The quality prediction module 300 is used to evaluate the weld quality. It inputs process parameters including the spindle speed, welding speed and shoulder pressure of the friction stir welding equipment and the weld point cloud model obtained from the three-dimensional reconstruction module 200 into a pre-trained convolutional neural network to evaluate the welding quality of the completed welded part, predict the tensile strength of the weld joint and whether there are defects in the weld. The welding quality prediction results output by the convolutional neural network are the tensile strength of the weld joint, the defect type and the evaluation index corresponding to the defect.

[0079] In this embodiment, the defect type of the weld output by the convolutional neural network is any one or more of the following: groove, hole, flash, and weld thinning.

[0080] The decision module 400 is used to determine whether the welding quality prediction result meets the welding requirements. If the welding quality prediction result meets the welding requirements, the current process parameters are maintained and welding continues. If the quality prediction result does not meet the welding requirements, a real-time adjustment plan is generated to adjust the process parameters.

[0081] The real-time adjustment scheme includes the following three scenarios:

[0082] A1: The tensile strength of the joint does not meet the welding requirements, but there are no defects in the weld: The decision module 400 adjusts the process parameters in real time until the welding requirements are met, and performs supplementary welding on the welds that do not meet the welding requirements.

[0083] A2: Welding defects exist in the weld: The decision module 400 eliminates the existence of defects by adjusting process parameters in real time, and matches the repair scheme for the defect by querying the database for the joint where the defect exists.

[0084] A3: For weld defects that cannot be repaired, stop the welding operation.

[0085] In this embodiment, for the joint where the defect exists, a repair solution is matched by querying a database. The repair solution includes:

[0086] (1) Groove defect repair scheme: If a groove defect occurs, match appropriate process parameters in the database according to the size, shape and location of the detected groove, including welding speed, rotation speed and pressure, as well as welding position and welding end position, etc. Then, repair the groove defect by welding to ensure the joint quality at the groove location.

[0087] (2) Hole defect repair scheme: If a hole defect appears in the weld, it can be repaired by filler welding. The repair scheme is based on the predicted hole size, shape and location. The filler material, welding speed, rotation speed and shoulder pressure are matched in the database to perform filler welding. A heat treatment scheme is also given to make the filler material bond with the base material.

[0088] The data transmission module 500 is used to access the friction stir welding CNC system to read and write data. By accessing the relevant data of each axis in the friction stir welding CNC system, it obtains process parameters. By calling the library functions of the dynamic link library encapsulated in the CNC equipment of the friction stir welding CNC system, it performs read and write operations on process parameters and controls the welding process.

[0089] In this embodiment, process parameters, including rotational speed, welding speed, and shoulder pressure, are obtained by accessing relevant data of each axis in the friction stir welding CNC system.

[0090] The machine vision-based method for monitoring the quality of friction stir welds in this embodiment, such as... Figure 3 As shown, before starting the quality monitoring of friction stir welds, preliminary preparations are required, including: powering on the friction stir welding equipment and initializing the friction stir weld quality monitoring system. After initializing the friction stir weld quality monitoring system, it is necessary to check whether the vision module has been calibrated. If the vision module has not been calibrated, it needs to be calibrated according to the calibration method of the binocular structured light measurement system. At the same time, the Modbus communication protocol is used to attempt to establish communication between the friction stir weld quality monitoring system and the friction stir welding CNC system. If the communication between the friction stir welding CNC system and the friction stir weld quality monitoring system fails, communication should be attempted again until communication is successfully established.

[0091] like Figure 4 As shown, the machine vision-based method for monitoring the quality of friction stir welds includes the following steps:

[0092] Step 1: The user sets the process parameters for friction stir welding according to the welding task, and performs stirring needle interference inspection on the friction stir welding equipment, the workpiece to be welded, the first array camera 1, the line laser projector 2, and the second array camera 3 to establish a reference plane.

[0093] The process parameters for friction stir welding include: spindle speed ω, welding speed υ, and shoulder pressure d.

[0094] Step 2: The friction stir welding equipment starts welding according to the set process parameters and acquires an image containing the laser stripe pattern on the weld surface.

[0095] In this embodiment, the user sets the process parameters for friction stir welding according to the welding task. After the parameter settings are completed, the friction stir welding equipment, the workpiece to be welded, and the first array camera 1, the line laser projector 2, and the second array camera 3 are subjected to stirring needle interference inspection. During the stirring needle interference inspection, the vision module 100 is used to acquire the three-dimensional point cloud data of the surface of the workpiece to be welded, and a reference plane Z=0 is established. The friction stir welding equipment starts welding with the set process parameters. At the same time, the first array camera 1, the line laser projector 2, and the second array camera 3 in the camera module 110 start working to acquire an image of the laser stripe pattern on the weld surface of the part that has been welded behind the welding spindle.

[0096] Step 3: Perform image preprocessing on the acquired image containing the laser stripe pattern on the weld surface, and extract the two sub-pixel precision light stripe center lines from the laser stripe pattern to obtain the three-dimensional point cloud data of the weld in the current state.

[0097] The image preprocessing includes: image filtering and denoising, image enhancement, and threshold segmentation.

[0098] In this embodiment, the image preprocessing process for the weld image is as follows: the acquired image containing the laser stripe pattern on the weld surface is sequentially subjected to image filtering and denoising, image enhancement and threshold segmentation to eliminate interference information and improve image quality and image processing efficiency.

[0099] In this embodiment, since the first array camera 1 and the second array camera 3 acquire grayscale images of laser stripe patterns with a certain width, it is necessary to extract the sub-pixel light stripe center lines in the laser stripe pattern. The extraction method used is as follows: the preprocessed laser stripe pattern is initially extracted using edge detection and geometric center method, the initially extracted pixel-level laser stripe center lines are quickly located, and then the normal of the initially extracted pixel-level laser stripe center lines is obtained using principal component analysis. An angular eight-neighborhood is divided at the center of the laser stripe, and the search for effective point sets is guided by the normal angle. The effective point set is then sub-pixel extracted using the normal centroid method to obtain two sub-pixel precision light stripe center lines. Based on the binocular stereo matching principle of line structured light, the three-dimensional coordinates of the two sub-pixel precision light stripe center lines projected on the weld surface are obtained, which is the three-dimensional point cloud data of the weld in the current state.

[0100] In this embodiment, different extraction methods can be used to extract the two sub-pixel precision center lines of the laser stripe pattern, such as the gray-scale centroid method, the Steger stripe center extraction method, and the curve fitting method. The gray-scale centroid method is a commonly used method for extracting the center of structured light stripes. Its principle is to calculate the laser stripe pattern row by row, using the abscissa of the gray-scale centroid of each calculated laser stripe as the center coordinate. The Steger stripe center extraction method based on the Hessian matrix determines the normal direction of the laser stripe pattern using the Hessian matrix, and then uses Taylor expansion along the normal direction to find the center position of the laser stripe pattern. This method has a large computational load and a slow extraction speed. The curve fitting method uses a Gaussian curve or parabola to fit the gray values ​​on the stripe cross section, and takes the abscissa of the extreme point after fitting as the center point of the light stripe.

[0101] Step 4: Construct a weld point cloud model based on the current state of the weld 3D point cloud data, segment the weld point cloud model to obtain the flash area point cloud model and the weld thinning area point cloud model, and calculate the volume U1 of the flash area point cloud model and the volume U2 of the weld thinning area point cloud model based on the reference plane.

[0102] Based on the motion characteristics of the friction stir welding equipment, the three-dimensional point cloud data of the weld in the current state is aligned to the world coordinate system to obtain the weld point cloud model. In this embodiment, the welding spindle moves in uniform linear motion on the XOY plane. According to the welding movement direction of the stirring head, the welding speed, and the shutter times of the first and second array cameras, the three-dimensional coordinates of the laser stripe center line at the current moment are unified to the world coordinate system to obtain the weld point cloud model at the current moment. Since the obtained weld point cloud model has measurement dead zones and shadow areas, there is a problem of missing point cloud data. The missing point cloud data is filled in using PCL library functions to obtain a complete weld point cloud model. Then, the weld point cloud model is segmented according to the spatial, geometric, and texture features of the weld to obtain the flash area point cloud model and the weld thinning area point cloud model. The volume of the flash area point cloud model and the volume of the weld thinning area point cloud model are calculated using the integration method.

[0103] The method for segmenting the weld point cloud model to obtain the flash region point cloud model and the weld thinning region point cloud model is as follows: In the weld point cloud model, the weld point cloud model is segmented based on the reference plane Z=0. The point cloud model composed of the three-dimensional point cloud data within the contour line formed by the points below the reference plane, i.e., the points Z<0, is denoted as the weld thinning point cloud model; the point cloud model composed of the three-dimensional point cloud data outside the contour line formed by the points above the reference plane, i.e., the points Z>0, is denoted as the flash point cloud model.

[0104] The method for calculating the volume U1 of the flash region point cloud model and the volume U2 of the weld thinning region point cloud model based on the reference plane is as follows: Based on the segmented flash region point cloud model and weld thinning region point cloud model, a triangular mesh of the weld point cloud model is constructed using the Delaunay triangulation algorithm. The projection of the triangular mesh onto the reference plane Z=0 is used as the base area, and the Zc coordinate of the centroid of the triangular mesh is used as the height. The volume of the flash region point cloud model and the volume of the weld thinning region point cloud model are solved by integration.

[0105] Step 5: Obtain the actual process parameters of friction stir welding at the current moment from the friction stir welding CNC system, and construct the actual process parameter matrix L. t .

[0106] In this embodiment, such as Figure 4 As shown, the friction stir welding CNC system is accessed via the Modbus communication protocol. A read motor pulse signal is sent to the friction stir welding CNC system. The friction stir welding weld quality monitoring system receives and parses the return signal from the friction stir welding CNC system, obtaining relevant data of the welding spindle, including: spindle speed, welding speed, and shoulder depressurization. The obtained data is used as the actual process parameters for friction stir welding in the current state, denoted as L. t :

[0107] L t :[ω t υ t d t ,t] (1)

[0108] Where L t This represents the actual process parameters of friction stir welding obtained via the communication protocol at time t; ω t υ represents the stirring head rotation speed at time t; t The welding speed of the stirring head at time t; d t This represents the amount of shoulder compression at time t.

[0109] Step 6: Convert the actual process parameter matrix L t By integrating the weld point cloud model with the actual process parameters, a 3D model of the weld is obtained.

[0110] In this embodiment, the actual process parameters L are used. t The weld point cloud model obtained in step 4 is fused with the actual process parameter L when the process parameters change at time t during the welding process. t It also changes accordingly, and L is updated according to formula (1). t Then update L tThe parameters are marked in the 3D model. It should be noted that the number of columns in the one-dimensional matrix of the actual process parameters is related to the number of process parameter items selected. The process parameter items are selected according to actual needs, and then the selected process parameter items are represented in one-dimensional matrix form. Then, this matrix is ​​fused with the 3D model according to the method described above.

[0111] Step 7: Use a pre-trained convolutional neural network to extract features from the weld 3D model with fused process parameters to obtain the joint tensile strength U3 of the weld and the evaluation index corresponding to the weld defect type. Then, combine the volume U1 of the flash area point cloud model and the volume U2 of the weld thinning area point cloud model to obtain the parameter matrix U characterizing the weld joint quality.

[0112] In this embodiment, the convolutional neural network adopts the VoteNet architecture and uses PointNet++ as the feature extraction network. It extracts features from the 3D weld model through four set abstraction (SA) layers and two frame protocol (FP) layers, taking the 1024 points from the last layer as the initial seed. Each seed has three dimensions of coordinates and C other features. The output of the convolutional neural network is combined with the volume U1 of the point cloud model of the flash region and the volume U2 of the point cloud model of the weld thinning region to obtain the final output parameter matrix U representing the quality of the weld joint, denoted as U = [U1, U2, U3, U4, U5]. T Where U1 represents the volume of the point cloud model of the flash region; U2 represents the volume of the point cloud model of the weld thinning region; U3 represents the tensile strength of the joint; U4 represents the groove evaluation index; and U5 represents the internal defect evaluation index. It should be noted that the evaluation indices corresponding to the weld defect types in the parameter matrix U, which characterizes the weld joint quality, can be replaced according to actual needs. The obtained parameter matrix U = [U1, U2, U3, U4, U5] T This is merely one expression in this embodiment.

[0113] Step 8: Determine whether the weld meets the welding requirements based on the parameter matrix U, which characterizes the weld joint quality. If it meets the welding requirements, continue welding while maintaining the current actual process parameters; otherwise, adjust U and L. t Input a pre-trained back propagation (BP) neural network to establish a real-time adjustment scheme and optimize the current real process parameters.

[0114] In this embodiment, such as Figure 5As shown, the process of establishing a real-time adjustment scheme is as follows: multiple BP neural networks with identical structures are used to represent friction stir welding operations, where each BP neural network corresponds to a parameter in the parameter matrix U that characterizes the weld joint quality, and L is respectively... t The parameters corresponding to each BP neural network are used as input to that BP neural network to obtain the shoulder underpressure amount corresponding to each parameter in the parameter matrix U, namely U1, U2, U3, U4, and U5 and their corresponding shoulder underpressure amounts. Generally, assigning each parameter to a separate BP neural network can improve the accuracy of the model, thereby reducing the number of hidden nodes and reducing the computational problems encountered in large networks. After obtaining the shoulder underpressure amounts corresponding to each parameter in matrix U, namely U1, U2, U3, U4, and U5 and their corresponding shoulder underpressure amounts, a genetic algorithm is used to search for the optimal shoulder underpressure amount, maximizing or minimizing the parameter corresponding to the optimal shoulder underpressure amount, keeping the other four parameters within the expected range, and supplementing the welding of the weld seams that do not meet the welding requirements.

[0115] Step 9: Control the friction stir welding equipment to continue welding according to the optimized actual process parameters of friction stir welding, and repeat steps 2-8 until the welding is completed.

[0116] In this embodiment, based on the actual process parameters of friction stir welding, the friction stir welding CNC system in the main body of the friction stir welding equipment is accessed through the Modbus communication protocol. The library functions of the dynamic link library encapsulated in the CNC equipment of the friction stir welding CNC system are called to perform read and write operations on the process parameters in the friction stir welding equipment and control the welding process. The optimized shoulder pressing amount is written into the friction stir welding CNC system, thereby realizing feedback control and defect repair welding functions.

[0117] In this embodiment, steps 2-8 are all displayed using 3D visualization.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A machine vision-based friction stir welding weld quality monitoring system, installed in a friction stir welding equipment, the friction stir welding equipment comprising a spindle, a stirring head, a workpiece, a worktable, and a friction stir welding CNC system, characterized in that, The monitoring system includes a vision module, a 3D reconstruction module, a quality prediction module, a decision-making module, and a data transmission module, which are electrically connected in sequence; the data transmission module is electrically connected to the friction stir welding CNC system. The vision module is used to acquire the laser stripe pattern projected onto the weld surface and send the acquired laser stripe pattern to the three-dimensional reconstruction module. The vision module further includes: a camera module for acquiring images containing laser stripe patterns on the weld surface; The camera module consists of a first array camera, a line laser projector, and a second array camera, which are fixed on the main shaft of the friction stir welding equipment according to the binocular vision model and the triangulation principle. The light plane of the line laser projector is coplanar with the baseline in the binocular vision model. The three-dimensional reconstruction module is used to draw a weld point cloud model based on the laser stripe pattern acquired by the vision module. The 3D reconstruction module further comprises three modules in sequence, namely: The image preprocessing module is used to analyze the characteristics of the images acquired by the first array camera and the second array camera, filter the noise in the images, and obtain the three-dimensional geometric information of the center line of the laser stripe pattern. The point cloud processing module is used to unify the three-dimensional geometric information of the center line of the laser stripe pattern into the world coordinate system; acquire the point cloud data of the weld and complete the point cloud data to obtain a complete weld point cloud model; segment the weld point cloud model to obtain the point cloud model of the flash area and the point cloud model of the weld thinning area. The calculation module is used to calculate the volume of the flash area point cloud model and the weld thinning area point cloud model based on the flash area point cloud model and the weld thinning area point cloud model obtained from the point cloud processing module, and to calculate the volume of the flash area point cloud model and the weld thinning area point cloud model using the integration method. The quality prediction module is used to evaluate weld quality. It integrates process parameters, including spindle speed, welding speed, and shoulder pressing amount, with the weld point cloud model to obtain a 3D weld model, which is then input into a pre-trained convolutional neural network to evaluate the welding quality of the completed welded parts and output the welding quality prediction results. The welding quality prediction results output by the convolutional neural network include the tensile strength of the weld joint, defect type, and the evaluation index corresponding to the defect. The decision module is used to determine whether the welding quality prediction result meets the welding requirements. If yes, the current process parameters are maintained and welding continues; if no, a real-time adjustment plan is generated to adjust the process parameters. The data transmission module is used to access the friction stir welding CNC system to read and write data. By accessing the relevant data of each axis in the friction stir welding CNC system, it obtains process parameters. By calling the library functions of the dynamic link library encapsulated in the CNC equipment of the friction stir welding CNC system, it performs read and write operations on process parameters and controls the welding process.

2. The machine vision-based friction stir welding weld quality monitoring system according to claim 1, characterized in that, The image preprocessing module further includes: The image filtering and denoising unit is used to denoise the image and eliminate interference information; The image enhancement unit is used to enhance the image and highlight the details in the image; The threshold segmentation unit is used to extract the region of interest (ROI) in the image and separate the laser stripe pattern on the weld surface from the background. The light stripe centerline extraction unit is used to extract the laser stripe pattern on the weld surface and obtain two light stripe centerlines with sub-pixel precision of the laser stripe pattern. The binocular stereo matching unit is used to calculate the positional deviation between corresponding points in the image and obtain the three-dimensional geometric information of the center line of the laser stripe pattern.

3. The machine vision-based friction stir welding weld quality monitoring system according to claim 2, characterized in that, The point cloud processing module further includes: The point cloud stitching unit is used to unify the acquired three-dimensional geometric information of the light stripe centerline into the world coordinate system to construct the weld point cloud model. The point cloud completion unit is used to repair missing point clouds in the weld point cloud model to obtain a complete weld point cloud model. The point cloud segmentation unit is used to segment the flash area point cloud model and the weld thinning area point cloud model from the complete weld point cloud model.

4. A machine vision-based method for monitoring the quality of friction stir welds, implemented using the machine vision-based friction stir weld quality monitoring system described in claim 3, characterized in that... The method includes the following steps: Step 1: The user sets the process parameters for friction stir welding according to the welding task, and performs stirring pin interference inspection on the friction stir welding equipment, the workpiece to be welded, the first array camera, the line laser projector and the second array camera to establish a reference plane. The first array camera, the line laser projector, and the second array camera are fixed on the main shaft of the friction stir welding equipment according to the binocular vision model and the triangulation principle, wherein the light plane of the line laser projector is coplanar with the baseline in the binocular vision model. Step 2: The friction stir welding equipment starts welding according to the set process parameters and acquires an image containing the laser stripe pattern on the weld surface; Step 3: Perform image preprocessing on the acquired image containing the laser stripe pattern on the weld surface, and extract the two sub-pixel precision light stripe center lines from the laser stripe pattern to obtain the three-dimensional point cloud data of the weld in the current state. Step 4: Construct a weld point cloud model based on the current 3D point cloud data of the weld. Segment the weld point cloud model to obtain the point cloud model of the flash area and the point cloud model of the weld thinning area. Calculate the volume of the flash area point cloud model based on the reference plane. Volume of the point cloud model of the weld thinning region ; Step 5: Obtain the actual process parameters of friction stir welding at the current moment and construct the actual process parameter matrix. ; Step 6: Convert the actual process parameter matrix By integrating the weld point cloud model, a three-dimensional weld model containing real process parameters is obtained; Step 7: Use a pre-trained convolutional neural network to extract features from the 3D model of the weld seam with fused process parameters to obtain the tensile strength of the weld joint. Evaluation indicators corresponding to weld defect types, combined with the volume of the point cloud model of the flash area. Volume of the point cloud model of the weld thinning region The parameter matrix U characterizing the quality of the weld joint is obtained; Step 8: Determine whether the weld meets the welding requirements based on the parameter matrix U characterizing the weld joint quality. If it meets the welding requirements, continue welding while maintaining the current actual process parameters; otherwise, merge the parameter matrix U characterizing the weld joint quality with the actual process parameter matrix. Input a pre-trained backpropagation neural network to establish a real-time adjustment scheme and optimize the current real process parameters; Step 9: Control the friction stir welding equipment to continue welding according to the optimized actual process parameters of friction stir welding, and repeat steps 2-8 until the welding is completed.

5. The method for monitoring the quality of friction stir welding seams based on machine vision according to claim 4, characterized in that, The image preprocessing described in step 3 includes: image filtering and denoising, image enhancement, and threshold segmentation.

Images