Weld quality visual monitoring method based on welding robot

Abstract

The present application relates to the technical field of image processing, and more particularly to a welding seam quality visual monitoring method based on a welding robot, which comprises the following steps: after the welding robot completes welding of a welding seam, continuously collecting images of the welding seam, aligning all initial welding seam images at a sub-pixel level to obtain a welding seam image sequence; obtaining suspected target connected domains in each welding seam image in the welding seam image sequence; performing dynamic trajectory correlation matching of suspected target connected domains on each two adjacent welding seam images to obtain at least one trajectory; modeling and quantifying a thermal decay behavior of each trajectory to obtain a comprehensive feature vector of each trajectory; and judging whether there is a defective trajectory according to the comprehensive feature vector of each trajectory by using a trained classification model, and determining a corresponding defect type when there is a defective trajectory, thereby reducing the interference of spatter on detection of porosity defects and improving the detection accuracy of welding defects.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a visual monitoring method for weld quality based on a welding robot. Background Technology

[0002] In modern intelligent manufacturing, welding robots are widely used in the production of critical structural components such as automobiles, ships, and pressure vessels. To ensure welding quality, vision sensors are usually integrated into the end effector of the robot to perform online quality inspection of the weld after welding. However, the welding environment is extremely complex, and transient interference generated during the welding process can affect the detection of real weld defects, especially tiny molten droplets (spatters). These spatters are between 0.1 and 2 millimeters in size and reach temperatures of thousands of degrees Celsius. After solidification, they adhere to the weld surface. In images acquired by vision sensors, these spatters appear as bright, nearly circular white spots, and their shape, size, and grayscale characteristics are almost indistinguishable from the most common weld defect—porosity (usually 0.5-3 millimeters in diameter).

[0003] Existing technologies typically rely on morphological filtering based on single-frame images to filter out splashes by setting size and shape thresholds. However, the actual size of splashes and pores highly overlaps, resulting in a very high false positive rate for morphological filtering methods based on single-frame images. When using multi-frame averaging or background subtraction to suppress transient noise, splashes that remain in the scene for multiple frames (such as when they are attached to a surface) will be retained, causing interference and making it difficult to accurately identify pore defects.

[0004] Therefore, how to distinguish between spatter and porosity in images and improve the detection accuracy of porosity defects in welding has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a visual monitoring method for weld quality based on a welding robot, in order to solve the problem of how to distinguish between spatter and porosity in images and improve the detection accuracy of porosity defects in welding.

[0006] This invention provides a visual monitoring method for weld quality based on a welding robot, the method comprising the following steps:

[0007] After the welding robot completes the welding, the weld seam is continuously image acquired to obtain the initial weld seam image at each sampling time. All the initial weld seam images are then aligned at the sub-pixel level to obtain the weld seam image sequence.

[0008] By constructing a background image and binarizing the image, the suspected target connected components in each weld image in the weld image sequence are obtained; by comprehensively considering the association cost, dynamic trajectory association matching of the suspected target connected components is performed on every two adjacent weld images to obtain at least one trajectory.

[0009] For any trajectory, based on the suspected target connected component corresponding to the trajectory, obtain the positional instability feature value; statistically analyze the sampling time sequence and gray value sequence corresponding to the trajectory, and obtain the trajectory lifetime based on the sampling time sequence; using the sampling time sequence and gray value sequence, perform nonlinear fitting on the exponential decay model used to describe the decay change law of trajectory gray value, and obtain the optimal cooling time constant, optimal initial amplitude and goodness of fit, and combine the positional instability feature value, trajectory lifetime, optimal cooling time constant, optimal initial amplitude and goodness of fit to form the comprehensive feature vector of the trajectory.

[0010] The trained classification model determines whether there is a defective trajectory based on the comprehensive feature vector of each trajectory, and when there is a defective trajectory, the corresponding defect type is determined.

[0011] Preferably, the step of obtaining the suspected target connected components in each weld image in the weld image sequence by constructing a background image and image binarization includes:

[0012] Based on the grayscale value of each pixel position in each weld image in the weld image sequence, the median of all grayscale values ​​at each same pixel position is obtained and recorded as the grayscale value at the corresponding same pixel position. Based on the grayscale value at each same pixel position, the background image is obtained.

[0013] For any weld image in the weld image sequence, calculate the absolute difference map between the weld image and the background image. Use a preset multiple of the grayscale standard deviation of the background image as the grayscale difference threshold. Use the grayscale difference threshold to binarize the absolute difference map to obtain the corresponding binarized image. The grayscale values ​​in the binarized image include a first preset value and a second preset value, and the first preset value is greater than the second preset value. Obtain the connected component composed of pixels with grayscale values ​​of the first preset value in the binarized image. Record the region in any weld image that is at the same position as the connected component as the suspected target connected component.

[0014] Preferably, the step of performing dynamic trajectory association matching of suspected target connected components on every two adjacent weld seam images by comprehensively considering the association cost, to obtain at least one trajectory, includes:

[0015] For the i-th suspected target connected component in the n-th weld image, calculate the comprehensive association cost between the i-th suspected target connected component and each suspected target connected component in the (n+1)-th weld image. Based on all comprehensive association costs, obtain the associated connected components of the i-th suspected target connected component in the (n+1)-th weld image and form a matching connected component pair.

[0016] Obtain all matching connected component pairs between any two adjacent weld seam images, and merge all matching connected component pairs that satisfy the transitivity of equivalence relations into a set; for any set, form a sequence of suspected target connected components in the set according to the temporal order of the weld seam images to which they belong, and record it as a trajectory corresponding to the set.

[0017] Preferably, the calculation of the comprehensive association cost between the i-th suspected target connected component and each suspected target connected component of the (n+1)-th weld seam image includes:

[0018] For the j-th suspected target connected region in the (n+1)-th weld image, obtain the Euclidean distance between the centroid position of the i-th suspected target connected region and the centroid position of the j-th suspected target connected region, and normalize the Euclidean distance to obtain the degree of positional deviation.

[0019] Obtain the gray-level histogram intersection coefficient between the i-th suspected target connected component and the j-th suspected target connected component, and record the difference between the constant 1 and the gray-level histogram intersection coefficient as the degree of gray-level difference;

[0020] The weighted sum of the positional deviation and the gray level difference yields the comprehensive association cost between the i-th suspected target connected component and the j-th suspected target connected component.

[0021] Preferably, the step of obtaining the positional instability feature value based on the suspected target connected component corresponding to any one of the trajectories includes:

[0022] Obtain the centroid position of each suspected target connected region corresponding to any trajectory, calculate the standard deviation of all centroid positions, and denote it as the position instability feature value.

[0023] Preferably, the exponential decay model is: ; in, This represents the grayscale value at the x-th sampling time in the sampling time sequence of the k-th trajectory. This represents the initial amplitude of the k-th trajectory. This represents an exponential function with the natural constant as its base. This represents the x-th sampling time in the sampling time sequence of the k-th trajectory. This represents the first sampling time in the sampling time sequence of the k-th trajectory. This represents the cooling time constant for the k-th trajectory. Let represent the steady-state offset constant of the k-th trajectory.

[0024] Preferably, the step of using the sampling time sequence and grayscale value sequence to perform nonlinear fitting on the exponential decay model used to describe the decay change law of trajectory grayscale, to obtain the optimal cooling time constant, optimal initial amplitude, and goodness of fit, includes:

[0025] Construct an objective function, and use an iterative optimization algorithm to obtain the optimal cooling time constant, optimal initial amplitude, and optimal steady-state offset constant corresponding to the minimum value of the objective function; substitute the optimal cooling time constant, optimal initial amplitude, and optimal steady-state offset constant into the exponential decay model to obtain the target exponential decay model; use the target exponential decay model to obtain the predicted grayscale value at each sampling time in the sampling time sequence of any trajectory to obtain the predicted grayscale value sequence; calculate the goodness of fit based on the predicted grayscale value sequence and the grayscale value sequence; wherein, the objective function is: ; in, This represents the objective function corresponding to the k-th trajectory. This represents the length of the sampling time sequence of the k-th trajectory.

[0026] Preferably, the gray value sequence refers to the sequence composed of the average gray values ​​of each suspected target connected component of any trajectory.

[0027] Preferably, the trajectory lifetime is the time interval between the first and last sampling moments in the sampling time sequence.

[0028] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0029] In this invention, in a scenario where a welding robot is continuously moving and capturing images, a weld image sequence is obtained by performing sub-pixel alignment on the continuously acquired initial weld images, ensuring stable tracking of grayscale changes of the same target. Then, from the weld image sequence, the appearance of the same suspected target in different frames is acquired and associated to form a complete spatiotemporal trajectory. Subsequently, the thermal attenuation behavior of each trajectory is modeled and its features are quantized to obtain a comprehensive feature vector composed of positional instability feature values, trajectory lifetime, optimal cooling time constant, optimal initial amplitude, and goodness of fit. This vector is used to characterize the dynamic thermal behavior and spatial morphological features of each trajectory. A highly robust classification model is trained using this comprehensive feature vector. The trained classification model and the real-time analyzed comprehensive feature vector are then used to detect welding defects and identify defect types, reducing the interference of spatter on porosity defect detection and improving the detection accuracy of welding defects. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart of a visual monitoring method for weld quality based on a welding robot, provided in Embodiment 1 of the present invention. Detailed Implementation

[0032] Embodiments of this disclosure are described in detail below, with examples of these embodiments illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting it.

[0033] It should be noted that the terms "first," "second," etc., used in this disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0034] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0035] See Figure 1 This is a flowchart of a visual monitoring method for weld quality based on a welding robot, provided in Embodiment 1 of the present invention. Figure 1 As shown, the method may include:

[0036] Step S101: After the welding robot completes the welding, the weld is continuously image acquired to obtain the initial weld image at each sampling time. All initial weld images are then aligned at the sub-pixel level to obtain a weld image sequence.

[0037] In actual welding robot operations, vision sensors are typically fixedly mounted on the end of the sixth axis of the welding robot or on an independent guide rail. After the welding robot completes welding a section of the weld, it moves to a preset detection position. At this point, the vision sensor begins scanning and imaging the weld, which is still at a high temperature, thus obtaining an initial weld image at each sampling moment. Due to the movement of the welding robot and possible mechanical vibrations, the position and orientation of the vision sensor relative to the weld are continuously changing when continuously acquiring images. Directly comparing pixels at the same location in different frames will lead to errors due to changes in viewing angle. Therefore, it is necessary to eliminate the influence of the welding robot's movement and obtain a sequence of weld images "similar to a still camera shot." The specific process is as follows:

[0038] The image acquisition is triggered immediately by the "welding complete" signal issued by the welding robot's controller, and a process lasting for a period of time is initiated. The high-speed image acquisition window utilizes a high-frame-rate global shutter industrial camera (such as the MV-CE200-10GM, resolution 1920×1200) to achieve high-speed image acquisition. Images of the weld were continuously acquired at a frame rate of [frame rate], and a total of [number] images were obtained. Initial weld image Each time the camera exposes a frame, a hardware trigger signal synchronously reads the six-dimensional pose data of the current end-effector center point (TCP) issued by the welding robot's controller. (Units: mm and degrees), where, This indicates the three-dimensional position of the center point of the robot's end effector in the base coordinate system of the welding robot. These pose data represent the rotation angles of the welding robot's end effector around each axis of the base coordinate system. These pose data describe the position and orientation of the camera's optical center relative to the welding robot's base coordinate system.

[0039] The first frame of the initial weld image Six-dimensional pose data of the camera during shooting The defined coordinate system serves as a reference coordinate system, used to transform the initial weld images of all frames to a coordinate system that is "as if in". The form of "shooting under pose". Specifically, through hand-eye calibration, the transformation matrix from the camera coordinate system to the welding robot end effector coordinate system is known. For the pose corresponding to the reference coordinate system Calculate the transformation matrix from the camera coordinate system to the base coordinate system of the welding robot. Similarly, for the first... The initial welding image is used to obtain the corresponding transformation matrix. Given the equation of the weld plane in the base coordinate system of the welding robot (which can be obtained through three-point calibration), establish the points from the initial weld image in frame 0. Points to the initial weld image in frame n The pixel-level mapping relationship is used for image registration, and the points of the initial weld image in frame 0. Points to the initial weld image in frame n There exists a homography transformation relationship between them: Among them, the homography matrix The following can be calculated using the camera pose and planar parameters corresponding to the two initial welding images: ; in, It is the camera intrinsic parameter matrix (obtained through calibration). and It is the rotation matrix and translation vector from the camera coordinate system corresponding to the initial welding image in frame 0 to the camera coordinate system corresponding to the initial welding image in frame n (which can be obtained from...). and (Derivation, existing technology, not detailed here) L is the unit normal vector of the world plane, represented in the camera coordinate system corresponding to the initial welding image in frame 0. It is the vertical distance from the camera's optical center to the world plane corresponding to the initial welding image in frame 0. This represents the transpose of the unit normal vector.

[0040] For the nth frame of the weld image sequence Each pixel coordinate The homography matrix obtained by the above calculation Find it in the first Initial weld image The corresponding pixel coordinates .because Typically sub-pixel coordinates, using bilinear interpolation from Obtain its grayscale value and assign it to Traverse the initial weld image in frame 0. The entire pixel range, i.e., obtaining the value of the first... Registration image with the initial weld seam image fully aligned This refers to the nth frame of the weld image. Similarly, the alignment and registration process described above is performed on each initial weld image frame to obtain a sequence of 100 strictly spatially aligned registered images, which is the weld image sequence. .

[0041] It should be noted that the aforementioned subpixel-level alignment is an existing technology and will not be elaborated upon here.

[0042] Step S102: By constructing a background image and binarizing the image, the suspected target connected components in each weld image in the weld image sequence are obtained; by comprehensively considering the association cost, dynamic trajectory association matching of the suspected target connected components is performed on every two adjacent weld images to obtain at least one trajectory.

[0043] After obtaining the weld seam image sequence, since any fixed physical point on the weld seam (such as a pore or a spatter attached to the surface) will appear at the exact same pixel position in the sequence, while a spatter flying in the air and gradually moving away will have a continuously changing pixel position in the weld seam image sequence because it does not meet the condition of being "fixed on the weld seam plane". Therefore, in this embodiment of the invention, the appearance of the same suspected target (suspected spatter) in different weld seam images is obtained and associated from the weld seam image sequence to form a complete spatiotemporal trajectory, thereby obtaining multiple trajectories formed by suspected dynamic spatter.

[0044] First, by constructing a background image and binarizing the image, suspected target connected components are obtained in each weld image in the weld image sequence, thus completing the acquisition of regions significantly brighter than the background. The specific process for identifying suspected target connected components is as follows:

[0045] The pixel position of each weld image in the weld image sequence Recorded as the same pixel position Based on the pixel position of each weld image in the weld image sequence The grayscale value is used to obtain the same pixel position. The median of all gray values ​​at a given location is recorded as the same pixel position. grayscale value at Similarly, the grayscale value at each same pixel location is obtained, and the background image is obtained based on the grayscale value at each same pixel location. Median filtering can effectively remove the influence of transient bright spots (splatter, defects) and obtain a clean weld background.

[0046] For any weld image in the weld image sequence, calculate the absolute difference map between the weld image and the background image, and record the pixel position in the absolute difference map. Difference at point A preset multiple of the grayscale standard deviation of the background image is used as the grayscale difference threshold. Preferably, the preset multiplier is set to 2. Then set the pixel position in the binarized image. The grayscale value is the first preset value; conversely, if... Then set the pixel position in the binarized image. The grayscale value is a second preset value, thereby obtaining the corresponding binarized image. The grayscale value in the binarized image includes a first preset value and a second preset value, and the first preset value is greater than the second preset value. Preferably, the first preset value is set to 1 and the second preset value is 0. In other embodiments, the first preset value can also be set to 255 and the second preset value is 0.

[0047] A connected component composed of pixels with a grayscale value of a first preset value is obtained from the binarized image. The region in any weld image that is at the same position as the connected component is recorded as a suspected target connected component. This yields the suspected target connected components in each weld image in the weld image sequence. Specifically, for the m-th suspected target connected component in the n-th weld image, the centroid position of the suspected target connected component is obtained. Average gray value and grayscale histogram .

[0048] Then, after obtaining the suspected target connected components in each weld image in the weld image sequence, inter-frame suspected target association is performed based on the features of each suspected target connected component. That is, by comprehensively considering the association cost, dynamic trajectory association matching of the suspected target connected components is performed on every two adjacent weld images to obtain at least one trajectory. The specific association matching method is as follows:

[0049] With the first Image of the first weld and the first Taking the image of the first weld as an example, let the first weld be... Each weld image has A suspected target connected component, denoted as... , No. Each weld image has A suspected target connected component, denoted as... For the i-th suspected target connected component in the n-th weld image and the j-th suspected target connected component in the (n+1)-th weld image, the Euclidean distance between the centroid positions of the i-th and j-th suspected target connected components is obtained. This Euclidean distance is normalized to obtain the degree of positional deviation. The gray-level histogram intersection coefficient between the i-th and j-th suspected target connected components is obtained, and the difference between the constant 1 and the gray-level histogram intersection coefficient is recorded as the degree of gray-level difference. The degree of positional deviation and the degree of gray-level difference are weighted and summed to obtain the comprehensive association cost between the i-th and j-th suspected target connected components.

[0050] The formula for calculating the comprehensive association cost between the i-th suspected target connected component and the j-th suspected target connected component is as follows: ; in, This represents the combined association cost between the i-th suspected target connected component and the j-th suspected target connected component. Indicates positional weight. This represents the centroid position of the i-th suspected target connected region in the n-th weld image. This represents the centroid position of the j-th suspected target connected region in the (n+1)-th weld image. This represents the normalization factor, which is 5% of the diagonal length of the weld image, and 1 represents a constant. This represents the gray-level histogram intersection coefficient between the i-th suspected target connected component and the j-th suspected target connected component. Let represent the gray-level histogram of the i-th suspected target connected component in the n-th weld seam image. This represents the grayscale histogram of the j-th suspected target connected region in the (n+1)-th weld seam image.

[0051] It should be noted that, The Euclidean distance between the centroids of two suspected target connected regions is calculated using... The Euclidean distance between the two centroid locations is normalized; therefore, using... The value of the penalty for excessive positional shift is related to the centroid position shift. When the centroid position shifts excessively, the corresponding positional shift cost is greater. For spatter or defects attached to the surface, their position in the welding image remains almost unchanged, so the corresponding positional shift cost is small. However, for spatter flying in the air, their position changes in different frames of the welding image, so the corresponding positional shift cost is large. The grayscale histogram intersection coefficient... This value is used to characterize the degree of grayscale overlap between two regions, i.e., the similarity. It ranges from 0 to 1; a higher value indicates a greater similarity in appearance between the two regions. This represents the degree of grayscale difference between the i-th suspected target connected component and the j-th suspected target connected component, used to penalize associations with excessive appearance changes.

[0052] Preferably, considering the overall association cost, positional consistency is given more weight; therefore, positional weights are set. No restrictions are imposed here.

[0053] Similarly, the comprehensive association cost between the i-th suspected target connected component and each suspected target connected component in the (n+1)-th weld seam image is obtained. A greedy nearest neighbor method is used to find the suspected target connected component with the minimum comprehensive association cost among the suspected target connected components in the (n+1)-th weld seam image, denoted as a candidate connected component. If the comprehensive association cost between the candidate connected component and the i-th suspected target connected component is less than a preset cost threshold of 0.3, then the candidate connected component is used as the associated connected component of the i-th suspected target connected component, forming a matching connected component pair with the i-th suspected target connected component. It should be noted that if an associated connected component of the i-th suspected target connected component cannot be found in the (n+1)-th weld seam image, the suspected target corresponding to the i-th suspected target connected component is considered to have disappeared in the (n+1)-th weld seam image, and the corresponding trajectory terminates. Similarly, suspected target connected components in the (n+1)-th weld seam image that are not matched are considered the starting point of new suspected targets, i.e., the starting point of new trajectories.

[0054] Following the method for obtaining the associated connected components of the ith suspected target connected component in the nth weld image, all matching connected component pairs between any two adjacent weld images are obtained. All matching connected component pairs that satisfy the transitivity of equivalence relations are merged into a single set. For example, if the ith suspected target connected component in the nth weld image and the j suspected target connected component in the (n+1)th weld image form a matching connected component pair, and the j suspected target connected component in the (n+1)th weld image and the z suspected target connected component in the (n+2)th weld image form a matching connected component pair, then the ith suspected target connected component, the j suspected target connected component, and the z suspected target connected component are grouped into one set. Thus, by processing 100 weld images, K sets are obtained, which represent K trajectories. Specifically: For any set, the suspected target connected components in the set are grouped into a sequence according to the temporal order of the weld seam images to which they belong, and this sequence is recorded as a trajectory corresponding to the set. For the k-th trajectory, the feature information of the k-th trajectory is obtained: the starting frame index of the trajectory. Observation data for each suspected target connected region within the trajectory's lifecycle, including but not limited to: the sampling time of the weld image to which the suspected target connected region belongs. Centroid location of the suspected target connected region Average gray value of suspected target connected components ,in , It is the number of suspected target connected regions contained in the k-th trajectory, which is also the number of weld images involved in the k-th trajectory. Each suspected target connected region contained in the k-th trajectory corresponds to a weld image.

[0055] Step S103: For any trajectory, obtain the position instability feature value based on the suspected target connected component corresponding to the trajectory; statistically analyze the sampling time sequence and gray value sequence corresponding to the trajectory, and obtain the trajectory lifetime based on the sampling time sequence; use the sampling time sequence and gray value sequence to perform nonlinear fitting on the exponential decay model used to describe the decay change law of trajectory gray value, and obtain the optimal cooling time constant, optimal initial amplitude and goodness of fit, and combine the position instability feature value, trajectory lifetime, optimal cooling time constant, optimal initial amplitude and goodness of fit to form the comprehensive feature vector of any trajectory.

[0056] Multiple trajectories were obtained through step S102. Each trajectory is treated as an independent "heat source," and its average gray value at different time points is... It can approximately reflect the temperature at the independent "heat source" (the camera response and thermal radiation intensity are linearly correlated within a certain range). Therefore, in this embodiment of the invention, thermal attenuation behavior is modeled for each trajectory, and its thermal attenuation behavior is quantified to extract the dynamic thermal behavior features and spatial morphological features of the trajectory for defect detection in subsequent classification models.

[0057] Taking the k-th trajectory as an example, assuming that the k-th trajectory is first detected in the 10th frame of the weld image and last detected in the 45th frame of the weld image, lasting for a total of 36 frames, the observation data of the extracted k-th trajectory is shown in Table 1 (time unit: seconds, grayscale value range: 0-255):

[0058] Table 1: Observation data of the k-th trajectory ;

[0059] Based on the observation data of the k-th trajectory, three sequences can be obtained: sampling time sequence. Gray value sequence Centroid position sequence The grayscale value sequence is a sequence composed of the average grayscale values ​​of each suspected target connected region of the k-th trajectory. Therefore, based on the centroid position sequence of the k-th trajectory, the first feature of the k-th trajectory can be obtained: the standard deviation of all centroid positions is calculated based on the centroid position of each suspected target connected region corresponding to the k-th trajectory, and denoted as the positional instability feature value. Specifically: Calculate the mean x-coordinate and mean y-coordinate of all centroid positions to form the average centroid position; calculate the Euclidean distance from each centroid position to the average centroid position; and record the standard deviation of all Euclidean distances as the positional instability characteristic value. In other embodiments, calculate the standard deviation of the x-coordinate and the standard deviation of the y-coordinate of all centroid positions, and combine the x-coordinate and y-coordinate standard deviations to obtain the composite standard deviation. ,in, The x-axis represents the standard deviation. The standard deviation of the ordinate is represented and denoted as the positional instability characteristic value. Simultaneously, based on the sampling time sequence of the k-th trajectory, the second characteristic of the k-th trajectory can be obtained: the time interval between the first and last sampling moments in the sampling time sequence is calculated and denoted as the trajectory lifetime of the k-th trajectory. In other implementations, the length of the sampling time sequence can also be... (The length of the sampling time sequence of the k-th trajectory is 36) is used as the trajectory lifetime of the k-th trajectory, which is not limited here.

[0060] Furthermore, based on Newton's law of cooling, a physical model of thermal decay is constructed. The cooling rate of a small-mass, high-temperature object in an environment approximately satisfies Newton's law of cooling: its temperature... With ambient temperature The difference decreases exponentially over time, that is ,in, It is the initial temperature. It is the cooling time constant. The smaller the value, the faster the cooling. Here, t represents time, and e represents the natural constant. Therefore, in weld images, grayscale values ​​are measured, which are not linearly related to temperature T. However, within the target temperature range, the camera response is approximately linear with the radiation intensity, so the grayscale difference should also exhibit an approximately exponential decay. Simultaneously, the background image... The gray level can approximate the radiation level at ambient temperature, thus allowing us to construct the following exponential decay model to describe the relative gray level decay of the k-th trajectory, i.e., the decay law of the trajectory gray level. The exponential decay model is as follows: ;

[0061] in, This represents the grayscale value at the x-th sampling time in the sampling time sequence of the k-th trajectory, which is also the x-th grayscale value in the grayscale value sequence. This represents the initial amplitude of the k-th trajectory. This represents an exponential function with the natural constant as its base. This represents the x-th sampling time in the sampling time sequence of the k-th trajectory. This represents the first sampling time in the sampling time sequence of the k-th trajectory. This represents the cooling time constant for the k-th trajectory. Let represent the steady-state offset constant of the k-th trajectory.

[0062] It should be noted that the initial amplitude Physically, this corresponds to the target's initial superheat (or contrast) relative to the background. For newly attached high-temperature splashes... The value is very large, which is in relation to the persistent defects. The value is relatively small; cooling time constant The smaller the value, the faster the cooling and the less splashing. Typically, the cooling time is on the order of tens to hundreds of milliseconds, while the cooling of defects, as part of the matrix, is synchronous with the overall weld and may last for several seconds or even longer. Within a 500ms observation window, the cooling time is approximately constant (i.e., the grayscale remains almost unchanged); steady-state offset constant. This represents the inherent grayscale difference that may still exist between the target and the background after the target has cooled to thermal equilibrium with the environment (e.g., due to differences in material color or oxidation). For a fully cooled splash, this refers to... It should be close to 0. For porosity defects, due to the scattering of light by the pit structure, It may be a non-zero constant.

[0063] Furthermore, the sampling time sequence of the k-th trajectory is used. and grayscale value sequence The least squares method was used to perform nonlinear fitting on the exponential decay model to obtain the optimal cooling time constant, optimal initial amplitude, and goodness of fit, specifically:

[0064] The sampling time sequence of the k-th trajectory and grayscale value sequence Substituted into the exponential decay model, used to estimate the parameters in the exponential decay model. Then, a nonlinear least squares method is used to construct the objective function, namely the sum of squared errors: ;

[0065] in, This represents the objective function corresponding to the k-th trajectory. This represents the length of the sampling time sequence of the k-th trajectory.

[0066] The objective function is obtained using an iterative optimization algorithm. The optimal parameter corresponding to the minimum value ,in, Let represent the optimal steady-state offset constant of the k-th trajectory, and the initial values ​​of the parameters are respectively... Then, the third and fourth features are extracted from the fitting results, namely: the optimal cooling time constant. This feature is the core feature; the smaller its value, the more likely it is a splash. Optimal initial amplitude. This feature is an auxiliary feature that reflects the initial contrast.

[0067] Finally, the optimal parameters Substituting into the exponential decay model, we obtain the target exponential decay model: ;

[0068] The sampling time sequence of the k-th trajectory Substituting into the target exponential decay model, and using the target exponential decay model, we obtain the predicted gray value at each sampling time in the sampling time sequence of the k-th trajectory, thus obtaining a gray value sequence. The corresponding predicted gray value sequence Furthermore, based on the predicted grayscale value sequence and the grayscale value sequence, the fifth feature of the k-th trajectory is obtained: goodness of fit, where goodness of fit... The method for obtaining it is: calculate the sum of squares of the grayscale value sequence. ,in, This represents the average gray value of the gray value sequence of the k-th trajectory. This represents the grayscale value at the x-th sampling time in the sampling time sequence of the k-th trajectory, which is also the x-th grayscale value in the grayscale value sequence. This represents the length of the grayscale value sequence or the length of the sampling time sequence; the sum of squared residuals is: , Represents the x-th predicted gray value in the predicted gray value sequence; goodness of fit Goodness of fit Used to quantify the ability of exponential decay models to explain data variation, ideally... , The closer the value is to 1, the stronger the exponential decay model's explanatory power for the data, and the more closely the grayscale changes in the observed data conform to the exponential decay law. For splashing: its cooling process physically approximates exponential decay, therefore the fitted exponential decay model can follow the changes in data points very well. It will be very small, usually leading to Very high (e.g.) For real defects, their temperature decreases slowly and synchronously with the matrix, and the grayscale may decrease approximately linearly or even remain almost unchanged within a short observation window. Forcibly fitting this pattern with an exponential decay model will result in a significant deviation between the predicted line and the data points, leading to… Larger. If Even greater than ,but It will be less than 0, negative. This is a strong indication that the exponential decay model is completely inapplicable to this trajectory, which in turn supports the judgment that "the target is not a rapidly cooled splash".

[0069] Thus, we have obtained five features of the k-th trajectory, namely, the positional instability feature value. trajectory lifespan Optimal cooling time constant Optimal initial amplitude and goodness of fit This is then combined into a comprehensive feature vector for the k-th trajectory. Similarly, following the method for obtaining the comprehensive feature vector of the k-th trajectory, the comprehensive feature vector of each trajectory is obtained.

[0070] Step S104: Using the trained classification model, determine whether there is a defective trajectory based on the comprehensive feature vector of each trajectory. If there is a defective trajectory, determine the corresponding defect type.

[0071] In the early stages of the laboratory or production line, a large amount of welding process data (including image sequences) is collected. Professional quality inspectors then precisely label the detected trajectories with defect types, categorizing them into "porosity," "cracks," "splashes," and "others." This primarily focuses on the binary classification problem of "defects" (porosity, cracks) and "splashes." For each labeled trajectory, its comprehensive feature vector is extracted following the steps described above. and category labels {Defects, Splashes}; Using these labeled datasets { A classification model is trained. Since the number of features is small (approximately 5 dimensions) and non-linear relationships may exist, a gradient boosting decision tree is used. This tree can automatically learn the interaction relationships between features and rank their importance, helping to verify whether they are indeed key features. After the classification model is trained, in a new welding task, the comprehensive feature vector of each trajectory is acquired in real time. The comprehensive feature vector of the k-th trajectory is input into the trained classification model, which outputs the probability that the k-th trajectory belongs to the "defect" type. .

[0072] Set a classification threshold, such as ,like If the condition is met, the k-th trajectory is determined to be a genuine defect; otherwise, the k-th trajectory is determined to be splash interference. Location: For trajectories judged as defects, the midpoint of their lifecycle (e.g., the k-th trajectory) is taken. The detection region contour corresponding to the frame is used as the precise pixel-level location of the defect in the image. This is combined with the robot pose corresponding to that frame. Based on camera calibration parameters, and using triangulation or known weld plane equations, the pixel coordinates of the defects are determined. Transform to 3D coordinates in robot base coordinate system This forms the basis for the robot to perform rework or marking. Based on the morphology of the defect area in the image, its equivalent diameter, aspect ratio, and area are calculated to help determine the defect type; for example, high roundness indicates pores, and a large aspect ratio indicates cracks. The system outputs a structured defect report, including: defect ID, type, confidence level, image location, 3D spatial location, size, and detection time. Simultaneously, the system can output statistical information on filtered spatter for process optimization.

[0073] It should be noted that using a trained classification model to determine whether there is a defective trajectory based on the comprehensive feature vector of each trajectory is an existing technology, which will not be elaborated here.

[0074] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A visual monitoring method for weld quality based on a welding robot, characterized in that, The method includes: After the welding robot completes the welding, the weld seam is continuously image acquired to obtain the initial weld seam image at each sampling time. All the initial weld seam images are then aligned at the sub-pixel level to obtain the weld seam image sequence. By constructing a background image and binarizing the image, the suspected target connected components in each weld image in the weld image sequence are obtained; by comprehensively considering the association cost, dynamic trajectory association matching of the suspected target connected components is performed on every two adjacent weld images to obtain at least one trajectory. For any trajectory, based on the suspected target connected component corresponding to the trajectory, obtain the positional instability feature value; statistically analyze the sampling time sequence and gray value sequence corresponding to the trajectory, and obtain the trajectory lifetime based on the sampling time sequence; using the sampling time sequence and gray value sequence, perform nonlinear fitting on the exponential decay model used to describe the decay change law of trajectory gray value, and obtain the optimal cooling time constant, optimal initial amplitude and goodness of fit, and combine the positional instability feature value, trajectory lifetime, optimal cooling time constant, optimal initial amplitude and goodness of fit to form the comprehensive feature vector of the trajectory. The trained classification model determines whether there is a defective trajectory based on the comprehensive feature vector of each trajectory, and when there is a defective trajectory, the corresponding defect type is determined.

2. The visual monitoring method for weld quality based on a welding robot according to claim 1, characterized in that, The step of obtaining the suspected target connected components in each weld image in the weld image sequence by constructing a background image and image binarization includes: Based on the grayscale value of each pixel position in each weld image in the weld image sequence, the median of all grayscale values ​​at each same pixel position is obtained and recorded as the grayscale value at the corresponding same pixel position. Based on the grayscale value at each same pixel position, the background image is obtained. For any weld image in the weld image sequence, calculate the absolute difference map between the weld image and the background image. Use a preset multiple of the grayscale standard deviation of the background image as the grayscale difference threshold. Use the grayscale difference threshold to binarize the absolute difference map to obtain the corresponding binarized image. The grayscale values ​​in the binarized image include a first preset value and a second preset value, and the first preset value is greater than the second preset value. Obtain the connected component composed of pixels with grayscale values ​​of the first preset value in the binarized image. Record the region in any weld image that is at the same position as the connected component as the suspected target connected component.

3. The visual monitoring method for weld quality based on a welding robot according to claim 1, characterized in that, The process involves performing dynamic trajectory association matching of suspected target connected components on every two adjacent weld seam images by comprehensively considering the association cost, resulting in at least one trajectory, including: For the i-th suspected target connected component in the n-th weld image, calculate the comprehensive association cost between the i-th suspected target connected component and each suspected target connected component in the (n+1)-th weld image. Based on all comprehensive association costs, obtain the associated connected components of the i-th suspected target connected component in the (n+1)-th weld image and form a matching connected component pair. Obtain all matching connected component pairs between any two adjacent weld seam images, and merge all matching connected component pairs that satisfy the transitivity of equivalence relations into a set; for any set, form a sequence of suspected target connected components in the set according to the temporal order of the weld seam images to which they belong, and record it as a trajectory corresponding to the set.

4. The visual monitoring method for weld quality based on a welding robot according to claim 3, characterized in that, The calculation of the comprehensive association cost between the i-th suspected target connected component and each suspected target connected component of the (n+1)-th weld seam image includes: For the j-th suspected target connected region in the (n+1)-th weld image, obtain the Euclidean distance between the centroid position of the i-th suspected target connected region and the centroid position of the j-th suspected target connected region, and normalize the Euclidean distance to obtain the degree of positional deviation. Obtain the gray-level histogram intersection coefficient between the i-th suspected target connected component and the j-th suspected target connected component, and record the difference between the constant 1 and the gray-level histogram intersection coefficient as the degree of gray-level difference; The weighted sum of the positional deviation and the gray level difference yields the comprehensive association cost between the i-th suspected target connected component and the j-th suspected target connected component.

5. The visual monitoring method for weld quality based on a welding robot according to claim 3, characterized in that, The step of obtaining positionally unstable feature values ​​based on the suspected target connected component corresponding to any one of the trajectories includes: Obtain the centroid position of each suspected target connected region corresponding to any trajectory, calculate the standard deviation of all centroid positions, and denote it as the position instability feature value.

6. The visual monitoring method for weld quality based on a welding robot according to claim 1, characterized in that, The exponential decay model is as follows: ； in, This represents the grayscale value at the x-th sampling time in the sampling time sequence of the k-th trajectory. This represents the initial amplitude of the k-th trajectory. This represents an exponential function with the natural constant as its base. This represents the x-th sampling time in the sampling time sequence of the k-th trajectory. This represents the first sampling time in the sampling time sequence of the k-th trajectory. This represents the cooling time constant for the k-th trajectory. Let represent the steady-state offset constant of the k-th trajectory.

7. The visual monitoring method for weld quality based on a welding robot according to claim 6, characterized in that, The step of using the sampling time sequence and gray value sequence to perform nonlinear fitting on the exponential decay model used to describe the decay change law of trajectory gray level, and obtaining the optimal cooling time constant, optimal initial amplitude and goodness of fit, includes: Construct an objective function, and use an iterative optimization algorithm to obtain the optimal cooling time constant, optimal initial amplitude, and optimal steady-state offset constant corresponding to the minimum value of the objective function; substitute the optimal cooling time constant, optimal initial amplitude, and optimal steady-state offset constant into the exponential decay model to obtain the target exponential decay model; use the target exponential decay model to obtain the predicted grayscale value at each sampling time in the sampling time sequence of any trajectory to obtain the predicted grayscale value sequence; calculate the goodness of fit based on the predicted grayscale value sequence and the grayscale value sequence; wherein, the objective function is: ； in, This represents the objective function corresponding to the k-th trajectory. This represents the length of the sampling time sequence of the k-th trajectory.

8. The visual monitoring method for weld quality based on a welding robot according to claim 3, characterized in that, The gray value sequence refers to the sequence composed of the average gray values ​​of each suspected target connected region of any trajectory.

9. The visual monitoring method for weld quality based on a welding robot according to claim 1, characterized in that, The trajectory lifetime is the time interval between the first and last sampling moments in the sampling time sequence.

Images