Laser welding control method

Abstract

This application applies to the field of laser welding technology, and in particular to a laser welding control method, comprising: obtaining an initial polarization image and an initial thermal image of a workpiece to be welded; determining a welding path and initial laser parameters for the workpiece to be welded based on the initial polarization image and the initial thermal image; controlling a laser welding device to weld the workpiece to be welded based on the welding path and the initial laser parameters, while simultaneously obtaining a real-time polarization image and a real-time thermal image of the workpiece to be welded; determining real-time status information of the molten pool of the workpiece to be welded based on the real-time polarization image, the real-time thermal image, and the welding path; determining real-time laser parameters based on the real-time molten pool status information, and controlling the laser welding device to weld the workpiece to be welded based on the real-time laser parameters. This method achieves precise welding process control, addressing the shortcomings of traditional vision systems.

Description

Technical Field

[0001] The present application belongs to the field of laser welding technology, and in particular to a laser welding control method. Background Art

[0002] Laser welding technology is widely used for joining metals. However, as the electronics, automotive, and medical industries continue to demand higher quality and efficiency in joining non-metallic materials such as thermoplastics, laser welding is increasingly being applied to non-metallic materials. Through techniques such as transmission laser welding, laser welding is being used to achieve efficient, precise, and low-heat-impact joining. This technology, which relies on the material's absorption and transmission properties of laser light and combines advances in optics, thermal science, and materials science, offers a new path for non-metallic welding.

[0003] Many non-metallic materials, such as plastics, ceramics, and glass, have strong surface reflectivity. Traditional vision systems typically rely on reflected light for imaging. However, for highly reflective materials, the reflected light can be too strong, resulting in overexposure or bright spots in the image, making it difficult to discern the actual location and shape of the weld.

[0004] Welds made of transparent or translucent non-metallic materials are difficult to clearly display through visible light reflection, resulting in traditional vision systems being unable to capture the correct welding area.

[0005] Compared with metal materials, many non-metallic materials have poor thermal conductivity. Traditional vision systems rely on surface image features to monitor the welding process and cannot directly capture the microscopic changes in the material caused by heat accumulation during the welding process.

[0006] In summary, when using traditional vision systems to control laser welding of non-metallic materials, there is a problem that traditional vision is difficult to capture the welds of non-metallic materials. Summary of the Invention

[0007] The embodiment of the present application provides a laser welding control method, which can solve the problem in the related art that when using a traditional vision system to control laser welding of non-metallic materials, it is difficult for traditional vision to capture the weld of non-metallic materials.

[0008] In a first aspect, an embodiment of the present application provides a laser welding control method, comprising:

[0009] Acquire an initial polarization image and an initial thermal image of a workpiece to be welded; wherein the workpiece to be welded is a non-metallic material;

[0010] determining a welding path and initial laser parameters of the workpiece to be welded according to the initial polarization image and the initial thermal imaging image;

[0011] Based on the welding path and the initial laser parameters, controlling the laser welding device to weld the workpiece to be welded, and simultaneously obtaining a real-time polarization image and a real-time thermal image of the workpiece to be welded;

[0012] Determining real-time status information of the molten pool of the workpiece to be welded based on the real-time polarization image, the real-time thermal image, and the welding path; wherein the real-time status information of the molten pool includes a molten pool offset and an average molten pool temperature;

[0013] According to the real-time status information of the molten pool, real-time laser parameters are determined, and according to the real-time laser parameters, the laser welding device is controlled to weld the workpiece to be welded.

[0014] The above technical solutions in the embodiments of the present application have at least the following technical effects:

[0015] The laser welding control method provided herein first obtains an initial polarization image and an initial thermal image of a workpiece (non-metallic material) to be welded. Based on the initial polarization image and thermal image, the welding path and initial laser parameters of the workpiece are determined. Based on the welding path and initial laser parameters, the laser welding device is controlled to weld the workpiece. Simultaneously, a real-time polarization image and thermal image of the workpiece are obtained. Based on the real-time polarization image, thermal image, and welding path, the real-time state information of the molten pool (molten pool offset and average molten pool temperature) of the workpiece is determined. Finally, based on the real-time molten pool state information, real-time laser parameters are determined and the laser welding device is controlled to weld the workpiece based on the real-time laser parameters. This method uses the initial polarization image and thermal image to determine the welding path and laser parameters of the workpiece to be welded, facilitating accurate target positioning and welding condition setting for the laser welding device from the outset, thereby achieving high-quality welding results. Using real-time polarization images and thermal images, the method dynamically monitors the molten pool state during the welding process, enabling timely detection of changes during the welding process. Laser parameters can then be adjusted based on real-time feedback to maintain welding stability and consistency. By continuously monitoring the molten pool and adjusting laser parameters based on real-time data, this method optimizes the molten pool's morphology and temperature distribution, avoiding overheating or uneven temperatures, thereby improving weld quality and reducing defects and deformation. This method not only enables precise welding process control but also addresses the shortcomings of traditional vision systems, providing more accurate weld tracking and real-time feedback. This significantly enhances the automation and intelligence of laser welding, improving the quality and efficiency of non-metallic material welding.

[0016] In a second aspect, an embodiment of the present application provides a laser welding control device based on machine vision, comprising:

[0017] An acquisition unit, configured to acquire an initial polarization image and an initial thermal image of a workpiece to be welded; wherein the workpiece to be welded is a non-metallic material;

[0018] an initial parameter determination unit, configured to determine a welding path and initial laser parameters of the workpiece to be welded based on the initial polarization image and the initial thermal imaging image;

[0019] a first control unit, configured to control the laser welding device to weld the workpiece to be welded based on the welding path and the initial laser parameters, and simultaneously obtain a real-time polarization image and a real-time thermal image of the workpiece to be welded;

[0020] a molten pool state monitoring unit, configured to determine real-time state information of the molten pool of the workpiece to be welded based on the real-time polarization image, the real-time thermal image, and the welding path; wherein the real-time state information of the molten pool includes a molten pool offset and an average molten pool temperature;

[0021] The second control unit is used to determine real-time laser parameters according to the real-time status information of the molten pool, and control the laser welding device to weld the workpiece to be welded according to the real-time laser parameters.

[0022] In a third aspect, an embodiment of the present application provides a machine vision-based laser welding control device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any one of the embodiments of the first aspect when executing the computer program.

[0023] It can be understood that the beneficial effects of the second to third aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 This is a flow chart of a laser welding control method provided in one embodiment of the present application;

[0026] Figure 2 This is a schematic diagram of the timing synchronization process of data acquisition before welding in the laser welding control method provided by an embodiment of the present application;

[0027] Figure 3This is a schematic diagram of the timing synchronization flow of data acquisition during welding in the laser welding control method provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0029] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0030] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0031] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0032] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0034] In related technologies, many non-metallic materials, such as plastics, ceramics, and glass, have strong surface reflectivity. Traditional vision systems typically rely on reflected light for imaging. Highlights created by the laser beam reflecting off the material surface can interfere with camera imaging, causing overexposure or bright spots in the image. This can lead to unclear weld areas and difficulty discerning the actual location and shape of the weld.

[0035] For transparent or translucent non-metallic materials (such as glass, transparent plastic, and certain composite materials), since these materials have weak absorption of the high energy of lasers, they often appear as irregular light spots, reflections, or glossy areas during welding. In addition, these materials may experience internal reflection or laser penetration, causing traditional vision systems to be unable to capture the correct welding area.

[0036] Many non-metallic materials have high gloss and irregular surface shapes, which are prone to uneven reflections. When the weld pool is generated during welding, surface deformation can obscure the weld edge or produce indistinguishable light spots. Traditional vision systems are not well suited to these complex glossy and irregular surface morphologies and are easily distracted by surface irregularities or reflected light, resulting in blurred images or inability to accurately distinguish welds.

[0037] Compared to metals, many non-metallic materials have poor thermal conductivity, leading to localized heat accumulation during laser welding and prone to problems such as overheating, cracking, or burn-through. Traditional vision technologies rely on surface image features to monitor the welding process, but they cannot directly capture the microscopic changes in the material caused by heat accumulation during welding. This makes welding defects such as localized overheating and lack of fusion difficult to detect, especially during welding processes with drastic temperature fluctuations.

[0038] To address the aforementioned issues, embodiments of the present application provide a laser welding control method. This method first obtains an initial polarization image and an initial thermal image of the workpiece to be welded (a non-metallic material). Based on these initial polarization images and thermal images, the welding path and initial laser parameters of the workpiece to be welded are determined. Based on the welding path and initial laser parameters, a laser welding device is controlled to weld the workpiece. Simultaneously, a real-time polarization image and a real-time thermal image of the workpiece to be welded are obtained. Based on these real-time polarization images, the real-time thermal image, and the welding path, real-time status information of the molten pool (molten pool offset and average molten pool temperature) of the workpiece to be welded is determined. Finally, based on this real-time molten pool status information, real-time laser parameters are determined, and the laser welding device is controlled to weld the workpiece based on these real-time laser parameters. This method uses the initial polarization image and thermal image to determine the welding path and laser parameters of the workpiece to be welded, facilitating accurate target positioning and welding condition setting for the laser welding device from the outset, thereby achieving high-quality welding results. This method uses real-time polarization images and thermal images to dynamically monitor the state of the molten pool during the welding process, allowing changes in the welding process to be captured in a timely manner. The laser parameters can then be adjusted based on real-time feedback to maintain welding stability and consistency. By continuously monitoring the molten pool state and adjusting the laser parameters based on real-time data, this method can optimize the morphology and temperature distribution of the molten pool, avoiding excessive heat treatment or uneven temperature, thereby improving welding quality and reducing defects and deformation. This method not only enables precise welding process control, but also compensates for the shortcomings of traditional vision systems, providing more accurate weld tracking and real-time feedback, significantly improving the automation and intelligence level of laser welding, and improving the quality and efficiency of non-metallic material welding.

[0039] The laser welding control method provided in the embodiment of the present application can be applied to a laser welding control device based on machine vision. In this case, the laser welding control device based on machine vision is the executor of the laser welding control method provided in the embodiment of the present application. The embodiment of the present application does not impose any restrictions on the specific type of the laser welding control device based on machine vision.

[0040] Exemplarily, the laser welding control equipment based on machine vision may include a polarization imaging device, a thermal imaging device, a laser welding device, and a control device that is communicatively connected to the polarization imaging device, the thermal imaging device, and the laser welding device. The polarization imaging device is a device that can capture the surface polarization image of a non-metallic material and can provide information about surface details, texture, smoothness, etc., and may include a multispectral polarization camera, a high-resolution polarization imaging camera, etc.; the thermal imaging device is a device that can capture the thermal image of a non-metallic material and can provide information related to temperature distribution, and may include a high-speed infrared thermal imager, an infrared thermal imaging camera, etc.; the laser welding device is a device that can perform laser welding on non-metallic materials, and may include a fiber laser welder, a CO2 laser welder, an ultraviolet laser welder, etc.; the control device is a device that can control the polarization imaging device, the thermal imaging device, the laser welding device, and can perform data processing, and may be a tablet computer, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), or a similar device. assistant, PDA), desktop computer, computing device or other processing device connected to a wireless modem, computer, laptop computer, etc.

[0041] In order to better understand the laser welding control method provided in the embodiment of the present application, the specific implementation process of the laser welding control method provided in the embodiment of the present application is exemplarily introduced below.

[0042] Figure 1 A schematic flow chart of a laser welding control method provided in an embodiment of the present application is shown. The laser welding control method includes:

[0043] S100: Acquire an initial polarization image and an initial thermal image of a workpiece to be welded, wherein the workpiece to be welded is a non-metallic material.

[0044] As you can understand, the purpose of a polarization imaging device is to capture polarized images of the surface of the workpiece to be welded. Polarized light can suppress specular reflections from non-metallic materials (such as smooth plastics and resin coatings on carbon fiber surfaces), enhancing the visualization of surface textures and detecting surface defects such as tiny cracks and pores. Polarization imaging devices can include multispectral polarization cameras and high-resolution polarization imaging cameras. A multispectral polarization camera can be used to acquire polarization images. A multispectral polarization camera can switch between polarization directions of 0°, 45°, 90°, and 135°, capturing reflected light at different polarization angles and providing rich surface information. An active polarization light source can utilize a circular LED array with an integrated linear polarizer. The light source angle can be adjusted to match the material's surface characteristics (e.g., an 850nm near-infrared light source can penetrate the carbon fiber surface).

[0045] Thermal imaging devices can capture thermal images of the workpiece being welded, monitor the workpiece's temperature distribution under low-power laser irradiation, and provide a basis for dynamic temperature control during subsequent welding. Thermal imaging devices can include high-speed infrared thermal imagers and infrared thermal imaging cameras. High-speed infrared thermal imagers (such as the FLIR X8580SC) are suitable options. They offer high frame rates (≥100Hz), high resolution (640×512), and a temperature measurement range covering welding temperatures (0-2000°C).

[0046] The purpose of a 3D scanning device is to acquire 3D point cloud data of the workpiece to be welded. This can include structured light 3D scanners, laser scanners, and other devices. Since non-metallic surfaces can have varying optical properties (e.g., smoothness or strong reflectivity), a structured light 3D scanner is suitable for non-metallic workpieces requiring high surface detail. It can quickly acquire 3D data over large areas and is highly adaptable to materials with highly reflective surfaces.

[0047] For example, before welding, the laser welding device has not yet started and cannot generate a pulse signal to trigger the simultaneous start of the polarization imaging device, thermal imaging device, and 3D scanning device. In this case, a PLC can be used to trigger the simultaneous start of the three devices. A command can be sent to the PLC via the host computer (i.e., the control device). The PLC is responsible for synchronously starting the three devices, which helps maintain consistent data acquisition time for all devices and avoids time misalignment due to response delays. To ensure time consistency in the acquisition of polarization images, thermal imaging images, and 3D point cloud data and avoid data misalignment due to device response delays, a high-precision clock source (such as a GPS synchronized clock) can be used to synchronize the three devices. A clock synchronization algorithm can be introduced into the host computer to ensure that the timestamp of each trigger signal is strictly aligned between the devices.

[0048] The polarization imaging device, thermal imaging device and three-dimensional scanning device are started synchronously by PLC control signal, such as Figure 2shown.

[0049] The polarization imaging device can use a polarized light source to illuminate the surface of the workpiece to be welded. Since the surface of non-metallic materials may have different surface roughness and structures, the polarization state of the reflected light will be affected. When receiving the reflected light, the multispectral polarization camera can use a polarization filter (such as a quarter-wave plate or polarizer) to separate polarized light in different directions. By changing the angle of the polarization filter, the reflected images at different polarization angles can be obtained. The camera can capture multiple images at different polarization angles and generate a polarization image of the surface of the workpiece to be welded by processing these images.

[0050] The non-local means (NLM) denoising algorithm can be used to remove noise from the polarization image, or wavelet threshold denoising can also be applied to remove higher frequency noise. The Stokes vectors (S0, S1, S2) can be extracted from the polarization image (0°, 45°, 90°, 135°). The Stokes vector can help quantify the information of different polarization states, that is, , , ,in, 、 、 、 are the light intensity measurements corresponding to the polarization angles. The degree of polarization (DoLP) can be calculated based on the Stokes vector, i.e. , DoLP describes the polarization intensity of light; the angle of polarization (AoP) is calculated based on the Stokes vector, that is AoP provides information about the polarization direction of light. Using radial and tangential distortion correction models, the polarization image can be calibrated based on the calibration results. This corrects the geometric distortion caused by the optical lens and ensures image accuracy. Following these steps, the polarization image captured by the polarization imaging device is preprocessed to produce an initial polarization image.

[0051] To avoid interference from ambient temperature, the workpiece to be welded can be kept at a stable ambient temperature. The thermal imaging device can compensate for the ambient temperature to eliminate the influence of external factors. At the appropriate distance and angle, the thermal imaging device scans the workpiece, capturing the thermal radiation information on the workpiece surface. During the scanning process, the processing system within the thermal imaging device converts the collected infrared radiation data into a thermal image, with each pixel representing the temperature of a specific point on the workpiece surface.

[0052] The thermal imaging device can be calibrated using a blackbody radiation source to establish a mapping table between grayscale values ​​and temperature. The temperature of the blackbody radiation source is known, and by measuring the radiation reflected back to the thermal imaging device, a standard grayscale-temperature conversion relationship can be established. Non-uniformity correction (NUC) can be used to improve the accuracy of the thermal image by eliminating differences in the response of the thermal imaging device's pixels (e.g., different temperature responses of different pixels). NUC calibrates and dynamically compensates for the responses of different pixels to ensure uniformity in the thermal image. Following these steps, the thermal image captured by the thermal imaging device is preprocessed to produce an initial thermal image. This initial thermal image provides baseline temperature distribution information for the workpiece before welding.

[0053] The structured light projector of a structured light 3D scanner projects a known light pattern onto the surface of the workpiece to be welded. This light pattern can be a simple stripe or a more complex grid or dot matrix. The irregular geometry of the workpiece surface causes the light pattern to distort. By shooting from multiple angles and projecting multiple light patterns, the scanner's camera captures these distorted patterns in real time. Based on the known light pattern and the data captured by the camera, the scanning system calculates the 3D coordinates of each surface point based on geometric relationships. All these 3D coordinates form a complete 3D point cloud, representing the shape and structure of the workpiece surface to be welded. Since the scanning process may capture some noise or inaccurate points, the point cloud data can be de-noised and optimized. If the scan is performed from multiple viewpoints, the point cloud data from different viewpoints can be subsequently registered and merged into a unified point cloud. Using surface reconstruction algorithms (such as Poisson reconstruction), the point cloud data can be converted into a continuous surface model, forming a complete 3D model of the workpiece to be welded. Structured light 3D scanners can efficiently and accurately acquire 3D point cloud data for non-metallic materials.

[0054] S200 , determining a welding path and initial laser parameters of the workpiece to be welded according to the initial polarization image and the initial thermal image.

[0055] For example, an image segmentation algorithm (such as a threshold method, GrabCut, etc.) can be used to divide multiple candidate welding areas on the initial polarization image or initial thermal image, and each candidate welding area can be scored. The scoring factors of the candidate welding area may include surface flatness, thermal stability, and material consistency. The surface flatness is determined based on the DoLP of the candidate welding area, that is, the higher the DoLP, the more suitable it is for welding; the thermal stability is determined based on the thermal image, that is, the more uniform the temperature distribution, the higher the thermal stability; and the material consistency is determined based on the AoP of the candidate welding area, that is, excessive material anisotropy makes it unsuitable for welding. A weighted sum is taken of the surface flatness, thermal stability, and material consistency of the candidate welding areas to calculate the score of each candidate welding area, and the candidate welding area with the highest score is selected as the final welding area.

[0056] If the workpiece to be welded has a pre-set weld line, the welding path can be extracted through image recognition (such as edge detection and Hough transform). If there is no pre-set weld line, the boundary of the final weld area can be selected and fitted into a straight or curved welding path. The welding path can be smoothed based on temperature and reflectivity gradients to avoid passing through high-risk areas. Topological analysis can also be used to determine the connectivity and closure of the welding path.

[0057] The welding path can be decomposed into a series of discrete path points, each of which represents the specific position of the laser. For each path point, the reflectivity of the point can be calculated based on the reflected light intensity and DoLP at different angles in the initial polarization image; the temperature change rate of the point can be calculated based on the pixel value in the thermal image, that is, ,in Indicates the The temperature change rate of the path point, Indicates the Temperature changes at path points (e.g., the difference between the first and second frames of thermal images) Temperature difference at the path points), Represents the time interval. The initial laser power of each path point is calculated based on the reflectivity and temperature change rate of each path point, that is, ,in, Indicates the Initial laser power at the path point, 、 is a constant that depends on the absorption characteristics of the material and the welding requirements, Indicates the The reflectivity of the waypoint.

[0058] The heat capacity of the workpiece to be welded can be calculated based on the material specific heat capacity and mass of the workpiece to be welded, that is, ,in, represents the heat capacity, represents the specific heat capacity, The initial laser velocity at each path point can be calculated based on the heat capacity and the temperature change rate of each path point, that is, ,in, Indicates the Initial laser velocity at the waypoint, is a constant that depends on the heat input required for welding and the thermal properties of the material.

[0059] If there is a structured light module or laser rangefinder, the distance from the surface of the workpiece to be welded to the laser head at each path point can be obtained in real time; if there is only a polarization image, the surface normal of each pixel can be restored based on DoLP and AoLP, and the distance from the surface to the laser head of each path point can be calculated by integration. The initial laser focal length of each path point is calculated based on the distance from the surface to the laser head, that is, ,in, Indicates the The initial laser focal length at the path point, Indicates the initial reference focal length (such as the focal length of the laser focus at the reference height). represents the distance from the surface to the laser head for each path point, Indicates the reference altitude.

[0060] The initial laser power, initial laser speed, and initial laser focal length of each path point are taken as initial laser parameters.

[0061] This step calculates the most suitable laser parameters for each welding path point, which facilitates precise control of each link in the welding process, thereby improving welding quality, optimizing the process and reducing manual intervention.

[0062] In one possible implementation, S200, determining a welding path and initial laser parameters of the workpiece to be welded based on the initial polarization image and the initial thermal image, includes:

[0063] S210 , performing weighted wavelet transform on the surface polarization degree of the initial polarization image and the baseline temperature gradient of the initial thermal imaging image to generate an initial fused image.

[0064] For example, a degree of polarization (DoLP) image (where surface polarization is present for each pixel) can be extracted from the initial polarization image using the same method as step S100 and will not be further described here. A baseline temperature gradient image can be calculated from the initial thermal image, which can be a spatial temperature gradient absolute value or directional magnitude image.

[0065] The size and coordinate system of the polarization degree image and the baseline temperature gradient image can be normalized (such as resampling, registration, and interpolation) so that the polarization degree and temperature gradient are within a unified grayscale range (such as 01 or 0255), which is conducive to comparability at the numerical level during fusion.

[0066] An appropriate wavelet basis can be selected based on fusion requirements. For example, the Daubechies (db4) wavelet is suitable for preserving detail intensity, while the Symlet (sym5) wavelet is more suitable for maintaining edge continuity. The degree of polarization image and the baseline temperature gradient image can each be subjected to n-layer wavelet decomposition, with each decomposition layer consisting of a low-frequency approximation subband (A) and three high-frequency detail subbands (horizontal H, vertical V, and diagonal D). The weights (α and β, with α + β = 1) for the degree of polarization image and the baseline temperature gradient image can be set based on the selected wavelet basis. If sym5 is used, α = 0.6 (for stronger polarization) and β = 0.4 can be set; if db4 is used, the opposite is true.

[0067] For each decomposition layer, the low-frequency approximate subbands and weights of the polarization degree image and the baseline temperature gradient image can be weighted averaged to obtain the low-frequency approximate subband after the two are fused. ; Use the maximum absolute value selection method (highlighting the edge) or the regional energy selection method to fuse the three high-frequency detail sub-bands of the two, namely ,in, is the fused horizontal high-frequency detail subband, is the horizontal high-frequency detail subband of the polarization image, is the horizontal high-frequency detail subband of the baseline temperature gradient image. The same is true for (vertical high-frequency detail sub-band after fusion), (Diagonal high-frequency detail subbands after fusion).

[0068] The fused coefficients 、 、 、 Perform n-layer inverse wavelet transform to restore a complete fused image. Histogram equalization or edge enhancement filtering can be performed on the fused image to enhance the weld structure features, filter out background noise or low response areas, and obtain an initial fused image with enhanced geometric edge structure and preserved temperature field characteristics.

[0069] This step fully preserves the structural details of the polarization image and the physical properties of the thermal image by organically combining polarization and thermal information in the wavelet domain, and achieves deep fusion in a multi-scale manner, greatly improving the accuracy of subsequent recognition and path planning.

[0070] S220 , extracting the geometric contour and material anisotropy of the weld region using a single-stream network based on the initial fused image to obtain a weld geometric feature vector and a material feature vector. The single-stream network may be a residual neural network or a convolutional neural network.

[0071] It can be understood that a single-stream network refers to a convolutional network structure that uses only one forward transmission path. It can be a convolutional neural network (CNN), such as VGG and MobileNet; it can be a residual neural network (ResNet), such as ResNet-18 and ResNet-34, which is used to enhance feature extraction capabilities and avoid gradient disappearance.

[0072] For example, the front-end of a single-stream network can identify edges, contours, and texture changes between the weld and the background in the initial fused image, extracting weld geometric boundary features and surface directional characteristics. The middle-end of the single-stream network can enhance perception of complex structures (weld discontinuities and blurred edges), allow skip-layer connections to prevent information loss in deep networks, and supplementally learn implicit features such as weak anisotropic textures and heat traces in the initial fused image. It outputs a multi-channel deep feature map, with each channel corresponding to a specific physical / geometric property (such as directional gradient and thermal energy distribution direction). The back-end of the single-stream network can apply global average pooling (GAP) or Flatten+fully connected layers to the deep feature map, mapping it into two types of feature vectors: weld geometric feature vectors (such as length, width change rate, average curvature, edge density, directional continuity, etc.) and material feature vectors (such as absorptivity, surface roughness, and thermal conductivity anisotropy).

[0073] This step can accurately extract the geometric contour features of the weld and the anisotropic characteristics of the material, providing precise and structured data support for subsequent welding path fitting and laser parameter setting, thereby significantly improving the accuracy of welding path planning and the adaptability of material response.

[0074] S230: Locate the starting point and the end point of the weld using a corner detection algorithm based on the initial fused image.

[0075] For example, corner detection aims to identify turning points or edge endpoints with strong local directional changes in an image. A corner detection algorithm (such as Harris corner detection, FAST corner detection, etc.) can be run on the initial fused image to obtain a series of corner coordinates (a two-dimensional pixel set), each with a response intensity value for sorting and screening.

[0076] The weld path can be fitted using contour extraction algorithms (such as contour tracing or skeleton extraction). All corner points are sorted along the weld direction, and the first and last two corner points are calculated based on the Euclidean distance or contour projection length, which are considered as the start and end points of the weld path.

[0077] If the general direction of the weld is known (such as from left to right), you can directly select the point with the smallest x coordinate as the starting point and the point with the largest x coordinate as the end point. For vertical or tortuous weld paths, you can select the first and last endpoints on the fitted path.

[0078] The coordinates of the two corner points obtained by this method represent the weld start and end points, and their positional accuracy can be visually verified in the image. If subsequent path fitting uses 3D coordinates, the pixel coordinates can be mapped to the 3D coordinates of the workpiece surface to be welded. This can be achieved by using a calibration matrix or projecting them into a 3D point cloud coordinate system.

[0079] This step uses image processing and corner point recognition technology to automatically extract key boundary points of the weld path, providing a spatial anchor point for subsequent welding path planning, trajectory fitting, and laser initial control.

[0080] S240 aligns the weld geometry feature vector, material feature vector, and weld start and end points with the three-dimensional point cloud data of the workpiece to be welded, fits the weld centerline, and generates a welding path for the workpiece to be welded. The three-dimensional point cloud data is obtained by scanning the surface of the workpiece to be welded using a three-dimensional scanning device.

[0081] For example, a rough weld path segment can be determined based on the projected 3D coordinates of the weld's start and end points, and a local point cloud region containing the weld can be cropped. Based on the weld's geometric eigenvector and material eigenvector, a set of spatial points (candidate point sets) that may constitute the weld path can be screened in the local point cloud region. The weld's geometric eigenvector is used to constrain the point cloud morphology of the weld region and guide the path search direction, while the material eigenvector is used to guide the selection of points in the point cloud with specific surface properties (such as normal changes and polarization response areas).

[0082] The candidate point set is sorted by spatial connectivity or directional continuity. A start-point-end-point guided search algorithm (such as the shortest path algorithm, RRT, or adaptive segmented fitting) can be used to extract a series of ordered spatial point sets along the weld direction. Three-dimensional B-spline curve fitting or Catmull-Rom spline interpolation can be used to smooth the ordered spatial point set and fit the weld centerline. If the weld path involves a 3D surface (such as a cylindrical surface weld), the fitted curve can be projected or fitted onto the point cloud surface.

[0083] The fitted weld center line can be used as the welding path of the workpiece to be welded, and the weld center point sequence and its tangential direction, normal direction, and material thermal response direction (for laser head posture planning) can be output synchronously.

[0084] The welding path generated by this step is not only geometrically continuous but also takes material properties into account, providing a solid foundation for a high-quality laser welding process.

[0085] S250, determining initial laser parameters according to the material characteristic vector.

[0086] For example, the material feature vector can be input into a trained multi-layer perceptron (MLP) model. The MLP model can output a set of initial laser parameters (laser speed, laser power, laser focal length / focus) to determine whether the output results fall within the safe operating range. If they exceed the range, they can be reasonably trimmed to ensure that the laser parameters are within the set range (such as laser power ∈ [600,1200] W). The final results can be transmitted to the control system for initial welding settings.

[0087] The training process of the MLP model: an actual laser welding test data set (sample data) can be collected. Each set of samples includes a material feature vector and the corresponding initial laser parameters (measured experimentally or set by the process engineer).

[0088] Build an MLP model: the input layer can be set to 12 neurons; the hidden layer can be set to 2 to 3 layers, with 64 or 128 neurons in each layer, and the activation function can be ReLU; the output layer can be set to 3 neurons (corresponding to power, speed, and focus respectively), and the activation function can be linear or restricted range Sigmoid mapped to the real number interval; a Dropout layer or L2 regularization can be added to prevent overfitting.

[0089] The model parameter weights can be initialized. In each round of iteration, forward propagation, loss calculation, backpropagation, and parameter update can be performed on each batch of sample data. After each round of iteration, the loss change is evaluated on the validation set. If the validation set loss does not decrease for N consecutive rounds, the early stopping mechanism is triggered, and the test set is used to evaluate the model accuracy. Evaluation indicators can include mean absolute error (MAE), mean relative error (MAPE), and individual regression accuracy of power, speed, and focus.

[0090] This step can intelligently infer the appropriate initial laser parameters from the actual photothermal response characteristics of the material, breaking away from the limitations of manual experience settings and improving welding quality and automation level.

[0091] In another possible implementation, S200, determining a welding path and initial laser parameters of the workpiece to be welded based on the initial polarization image and the initial thermal image, includes:

[0092] S201: Extracting a texture feature vector using a polarization flow of a first dual-stream network based on the surface polarization degree and surface polarization angle of the initial polarized image, wherein the polarization flow of the first dual-stream network includes a residual neural network and an attention mechanism.

[0093] For example, the degree of polarization (DoLP) image and angle of polarization (AoP) image (each pixel has a surface degree of polarization and a surface angle of polarization) can be extracted from the initial polarization image. The two images serve as complementary inputs, reflecting the information of intensity changes and direction changes respectively. For unified processing, the two images can be stacked into a two-channel input tensor.

[0094] It can be understood that the polarization flow is a sub-pathway in the first dual-stream network, which is dedicated to extracting surface texture feature vectors from polarization images. The residual neural network can be responsible for extracting low, medium and high-level features such as the spatial structure, texture density, and edge frequency of the surface texture. The attention mechanism can more effectively identify key texture areas and areas with significant surface direction changes. The attention module can be embedded in the middle and back of the residual structure to enhance the output feature map.

[0095] For example, a residual neural network (such as ResNet-18) can be used to feed a dual-channel input tensor into ResNet-18. After processing through several residual blocks, multi-scale and multi-level spatial texture features are extracted, resulting in a polarization texture feature map. Each residual block can include two convolutions, batch normalization, activation (ReLU), and residual skip connections. The number of output channels gradually increases, while the spatial resolution gradually decreases.

[0096] The attention mechanism can be a channel attention mechanism (SE module), which automatically assigns a weight to each feature channel, highlighting the direction, edge, and frequency features that are meaningful for texture discrimination. For example, it emphasizes areas with anisotropy on the surface and deemphasizes flat background areas. It can also be a spatial attention mechanism (CBAM or SAM), which locates salient texture areas in the polarization texture feature map, such as boundaries, fiber intersections, or corners, further strengthening the network's ability to focus on key areas and suppressing ineffective areas. Either the channel attention mechanism or the spatial attention mechanism can be used independently, or both can be used to form a hybrid attention mechanism. The output image retains the same shape, but important texture information is enhanced and irrelevant areas are suppressed in the polarization texture feature map.

[0097] After processing by the residual network and attention mechanism, the first two-stream network obtains a set of high-dimensional feature maps, which can be compressed into a fixed-length high-dimensional vector (texture feature vector) through the global average pooling (GAP) operation.

[0098] This step realizes the automatic conversion from low-level optical information to high-level material physical information. It can be applied to the surfaces of complex materials such as non-metals. It can obtain real process characteristics from microscopic directionality and texture differences, and provide quantifiable expression for multimodal information fusion.

[0099] S202: Extracting a temperature feature vector based on the temperature field of the initial thermal image using the heat flow of a first dual-stream network, wherein the heat flow of the first dual-stream network includes a residual neural network and a temporal convolutional network.

[0100] For example, the initial thermal image can be expanded into pseudo time series data through a sliding window or simulation time step construction method, that is, a single-frame thermal image is copied into multiple scale views or its thermal gradient diffusion morphology is simulated to generate a multi-scale temperature map sequence.

[0101] Each frame of the multi-scale temperature map sequence can be input into ResNet-18, and spatial gradient features (such as heat distribution contours, boundary structures of high and low temperature areas, and temperature gradient mutation lines) can be extracted through multiple residual blocks to generate a temperature feature map for each frame.

[0102] With time as the first dimension, the temperature feature maps of all frames can be stacked into a sequence of temperature feature maps. This sequence is then fed into a temporal convolutional network (such as a 1D temporal convolution). This network processes the temperature trends of each channel over time or scale along the T-axis, similar to a sliding window in the time dimension. It learns the changing patterns of temperature responses at each spatial location and generates a temporal feature map that integrates spatial and thermal diffusion trends. Compared to RNNs / LSTMs, temporal convolutional networks (TCNs) offer greater parallelism and are suitable for spatial and thermal data processing.

[0103] Global average pooling (GAP) or adaptive pooling can be performed on the time series feature map to form a temperature feature vector that globally expresses the thermal behavior of the material.

[0104] This step can effectively extract the thermal response characteristics of the workpiece to be welded under the initial thermal field by combining the extraction of temperature features by the residual neural network and the modeling of heat diffusion trends by the temporal convolutional network, thereby providing deep, physically meaningful temperature feature support for welding path planning and laser parameter setting, and achieving more accurate and adaptive welding control.

[0105] S203 , performing vector fusion on the texture feature vector and the temperature feature vector to obtain a first fused feature vector.

[0106] For example, the texture feature vector and the temperature feature vector can be fused using vector concatenation, weighted fusion, or fusion network mapping (such as interactive mapping of two vectors using a small MLP or attention layer) to generate a single global expression (the first fused feature vector).

[0107] S204: Generate a welding path of the workpiece to be welded based on the first fused feature vector and the three-dimensional point cloud data.

[0108] For example, the texture and heat diffusion direction features in the first fused feature vector can be used to identify areas on the point cloud surface most likely to contain welds. For example, if the first fused feature vector contains linear strips of heat concentration and a texture in a specific direction, a corresponding continuous strip-like region is searched in the 3D point cloud to form a candidate region.

[0109] Based on the surface normal variations, curvature continuity, and directional consistency of the point cloud in the candidate region, a series of center point sequences can be extracted to represent the weld's central axis orientation. Extraction methods can include local normal voting (to determine edge orientation) and local curvature minimum path search. The center point sequence can be smoothed using 3D spline fitting or Catmull-Rom curves to produce a spatially smooth and continuous weld centerline trajectory (welding path). The welding path can be sampled at equal intervals to produce a sequence of path points, each of which includes the spatial coordinates, as well as the normal and tangent directions at that point.

[0110] This step can achieve adaptive coupling between path and material properties, improve welding consistency, and provide high-quality spatial basic data for subsequent laser trajectory control and closed-loop adjustment.

[0111] S205: Determine initial laser parameters according to the first fused feature vector.

[0112] Exemplarily, the MLP model can also be used, and each group of samples in the sample data of step S250 is replaced with the first fused feature vector and the corresponding initial laser parameters. The training process is similar.

[0113] The first fused feature vector is input into the trained MLP model, and the MLP model can output a set of initial laser parameters (laser speed, laser power, laser focal length / focus).

[0114] This step can generate highly adaptive laser parameters, eliminating the need for manual trial welding, improving the first-time welding success rate and reducing material loss.

[0115] S300, based on the welding path and the initial laser parameters, controlling the laser welding device to weld the workpiece to be welded, and simultaneously obtaining a real-time polarization image and a real-time thermal image of the workpiece to be welded.

[0116] For example, the welding path and the initial laser power, initial laser speed, and initial laser focal length of each path point can be organized into an internal control data structure (such as a task queue, a parameter table, and an instruction cache). For each path point, a set of control instructions is generated, which may include a movement instruction for instructing the laser head of the laser welding device to move to , the speed is ; Laser power setting instruction, used to set the output power of the laser welding device to ;Focus adjustment command, used to set the focus mechanism of the laser welding device to adjust the focal length Laser trigger instructions are used to control the laser welding device to emit a laser beam at the set power at the current position. These instructions can be sent to each control module using a standard motion control language (such as G-code) or via a dedicated industrial bus (such as EtherCAT). Specifically, movement instructions are sent to the motion control module (for path point positioning and speed control), laser power setting instructions and laser trigger instructions are sent to the laser control module (for setting laser power and turning the laser beam on / off), and focal length adjustment instructions are sent to the focus control module (for driving the electric lens or mirror group to complete focal length adjustment). The controller coordinates all control modules to operate synchronously according to the instructions, ensuring that the laser illuminates the workpiece at the exact position and with the set parameters.

[0117] In welding, the pulse signal output by the laser can be used to trigger the polarization imaging device and the thermal imaging device to collect images, achieving microsecond time control, such as Figure 3 shown.

[0118] It can be assumed that each welding pulse output by the laser welding device is the master trigger event, and the image acquisition system (polarization imaging device and thermal imaging device) is a slave system, executing with a fixed delay. That is, each laser pulse emits a TTL trigger signal, which is sent to the polarization imaging device. Upon receiving the TTL trigger signal, the polarization imaging device immediately triggers the multispectral polarization camera to take an exposure. The exposure time can be set to 100us, with the exposure start time being t and the exposure end time being t+100us. The real-time polarization image reflects the surface reflectivity very shortly after the laser is applied. After the polarization exposure starts, a high-precision timer can be used to delay the time by 200us. At t+200us, the high-speed infrared thermal imager is triggered to take an exposure. The exposure time can be set to 100us, resulting in an acquisition time of t+200us to t+300us. The thermal image captures the early thermal diffusion state after laser heating. Closely following the laser pulse can capture changes in the reflective surface. Delaying the triggering of the thermal imaging device prevents laser interference with the infrared lens while allowing the thermal response to form a measurable signal.

[0119] This step realizes the precise control of laser welding based on the welding path and initial laser parameters. Through a timing mechanism synchronized with the laser pulse, real-time polarization images and thermal images of the welding process are collected with high efficiency and low latency, thereby obtaining synchronized visual information of each path point in the welding process, providing high-quality and high-precision basic data support for subsequent molten pool status monitoring and adaptive adjustment of laser parameters.

[0120] S400: Determine real-time status information of the molten pool of the workpiece to be welded based on the real-time polarization image, the real-time thermal image, and the welding path. The real-time status information of the molten pool includes a molten pool offset and an average molten pool temperature.

[0121] For example, for each path point: the pixel value in the real-time thermal imaging image can be converted into the actual temperature (such as the temperature corresponding to the (50th, 120th) pixel is 812.6°C) through emissivity compensation and the camera's built-in temperature conversion model (or lookup table), and a temperature threshold can be set, and the area greater than the temperature threshold is determined as the first molten pool area.

[0122] The degree of polarization (DoLP) at each pixel position can be calculated based on the real-time polarization image. The calculation method is the same as step S100. A DoLP threshold can be set, and an area smaller than the DoLP threshold is determined as the second molten pool area.

[0123] The intersection of the first and second molten pool areas can be used as the final molten pool area. The coordinates of the molten pool center can be calculated based on each pixel coordinate of the final molten pool area (the real-time polarization image and the real-time thermal image have completed image registration and alignment, and the pixel coordinates in the two are the same) and the corresponding temperature value, that is, , ,in, represents the coordinates of the center of the melt pool, Represents the final molten pool area Each pixel coordinate in Represents the temperature value of each pixel in the thermal image.

[0124] The pixel coordinates of the weld center in the image can be calculated based on the welding path, and the molten pool offset can be calculated based on the pixel coordinates of the weld center and the coordinates of the molten pool center, that is, ,in, Indicates the melt pool offset, The pixel coordinates representing the center of the weld.

[0125] The average temperature of the molten pool can be calculated based on the number of pixels in the final molten pool area and the temperature value of each pixel, that is, ,in, represents the average temperature of the molten pool, The number of pixels representing the final melt pool area.

[0126] This step realizes the fusion analysis based on thermal imaging, polarization images and welding paths, and calculates the molten pool offset and average molten pool temperature of each path point in real time, providing key perception basis for the subsequent closed-loop adjustment of laser parameters, and improving the stability and intelligence of the welding process.

[0127] In one possible implementation, S400 determines the real-time status information of the molten pool of the workpiece to be welded based on the real-time polarization image, the real-time thermal image, and the welding path, including:

[0128] S410 , performing wavelet decomposition on the molten pool polarization degree of the real-time polarization image and the molten pool temperature gradient of the real-time thermal imaging image, and weightedly fusing the low-frequency and high-frequency components of the molten pool polarization degree and the molten pool temperature gradient to generate a real-time fused image.

[0129] Exemplarily, the method for generating the real-time fused image is the same as step S210 and will not be described again here.

[0130] This step can enhance the temperature field characteristics while maintaining the structural details of the molten pool, generating a real-time fusion image that combines spatial texture and heat diffusion information, thereby improving the accuracy and robustness of subsequent molten pool feature extraction and providing more comprehensive and stable visual input for welding state perception and control.

[0131] S420 , based on the real-time fusion image, a single-stream network is used to extract low-level features and high-level features of the molten pool to obtain a comprehensive feature vector.

[0132] For example, the first few layers of the single-stream network (such as small-size convolution kernels (such as 3×3)) can be used to extract low-level features of the melt pool (such as local edges, melt pool contours, temperature contrast and other detailed features) to obtain low-level feature maps; the latter layers of the single-stream network (multi-layer convolution and pooling) can be used to extract global features (high-level features) with abstract semantics to obtain high-level feature maps, such as the overall shape and expansion direction of the melt pool, the heat distribution trend in the center of the melt pool, the surface directional changes and the energy concentration area.

[0133] Skip connections and multi-scale splicing can be used to fuse low-level feature maps with high-level feature maps, perform global average pooling (GAP) on the fused feature maps, compress the two-dimensional spatial information into a one-dimensional numerical vector, and obtain a fixed-length comprehensive feature vector (such as 64 dimensions, 128 dimensions, 256 dimensions, etc.).

[0134] This step compresses the complex information in the multimodal image into quantifiable and traceable feature vectors, improving the system's ability to identify, classify and adjust the melt pool state in real time.

[0135] S430, calculating the molten pool offset based on the real-time fusion image and the welding path, and calculating the average molten pool temperature based on the comprehensive feature vector.

[0136] As you can understand, the molten pool offset represents the horizontal or vertical deviation between the actual center of the current molten pool and the ideal welding path, and is used to determine whether the weld has deviated from the target trajectory. The average molten pool temperature is a key factor in controlling parameters such as laser power and welding speed, and it reflects the current welding heat input status.

[0137] For example, in the real-time fusion image, the molten pool area can be extracted by threshold method, edge detection or segmentation algorithm (such as based on heat value or texture intensity) to obtain a molten pool area mask, and based on the molten pool area mask, the center of mass (i.e., area-weighted center) of the molten pool area is calculated, which is the current molten pool center coordinate of the workpiece to be welded.

[0138] The spatial coordinates of the current path point can be determined based on the path point information in the welding path. Since the welding path is in three-dimensional space, the spatial coordinates of the current path point can be projected into image coordinates using calibration parameters, or the current weld pool center coordinates can be back-projected into the welding space to ensure that the current weld pool center coordinates and the coordinates of the current path point are in the same reference system. The difference between the current weld pool center coordinates and the coordinates of the current path point is calculated as the weld pool offset.

[0139] Based on a trained regression model (such as MLP, random forest regression, or linear regression), the comprehensive feature vector can be mapped to the average melt pool temperature (regression value). If an MLP model is used, each sample group in the sample data of step S250 can be replaced with the comprehensive feature vector and the corresponding average melt pool temperature (which can be measured by a thermal imager or derived from laser power and material model). The training process is similar. Alternatively, the melt pool region can be extracted from the real-time fused image and the average temperature of the melt pool region calculated as an estimated value. This estimated value and the regression value are then weighted and summed to improve stability.

[0140] This step can convert the real-time fusion image into quantifiable physical state quantities of the molten pool, realizing a closed-loop bridge from visual perception to welding control. It can calculate the offset and temperature of the molten pool with high precision, adapt to different materials and surface conditions, and provide key real-time feedback parameters for the fully automatic laser welding system.

[0141] In another possible implementation, S400 determines the real-time state information of the molten pool of the workpiece to be welded based on the real-time polarization image, the real-time thermal image, and the welding path, including:

[0142] S401: Extracting a melt pool texture feature vector using the polarization stream of a second dual-stream network based on the melt pool polarization degree of the real-time polarization image. The polarization stream of the second dual-stream network includes a MobileNetV3 and a long short-term memory network.

[0143] MobileNetV3 is a lightweight convolutional neural network with excellent feature representation and computational efficiency, making it suitable for deployment in real-time welding systems. Because the molten pool is a dynamically changing region, accurately identifying its trends using static frames alone is difficult. Long Short-Term Memory (LSTM) networks capture temporal correlations between consecutive frames and identify texture changes. Combining MobileNetV3 with LSTM can capture the spatial texture structure and dynamic trends of the molten pool during the welding process.

[0144] Exemplarily, the polarization image of the previous T frames of the real-time polarization image is obtained, and the polarization degree images of the real-time polarization image and the polarization image of the previous T frames can be extracted. All polarization degree images are input into MobileNetV3. The network extracts local and global texture patterns in all polarization degree images through multiple depth-separable convolutional layers and nonlinear activation modules, which may include the edge structure of the melt pool, internal polarization symmetry, surface disturbance texture and stability, thereby obtaining a polarization degree feature image sequence.

[0145] Global average pooling (GAP) is performed on each frame image in the polarization feature image sequence to generate a feature vector sequence. The feature vector sequence is input into the LSTM for temporal modeling. The LSTM can capture the temporal texture changes and output the hidden state vector at the final moment as the melt pool texture feature vector, which integrates the spatial structure and time dimension evolution of the current frame (real-time polarization image).

[0146] S402: Extracting a melt pool temperature feature vector based on the temperature field of the real-time thermal image using the heat flow of a second dual-stream network, wherein the heat flow of the second dual-stream network includes a residual neural network and gradient convolution.

[0147] For example, you can extract the raw thermal grayscale matrix from a real-time thermal image (or obtain the actual temperature matrix through the SDK / API) to obtain a temperature field data matrix (where the value of each pixel represents the temperature or normalized temperature at that location). A temperature field is essentially a single-channel image (temperature image) where each pixel value represents temperature.

[0148] Residual neural networks (ResNet) can extract the spatial temperature distribution features of a temperature image through multiple residual blocks, generating a multi-channel temperature spatial feature map. A custom gradient convolution kernel (such as a Sobel-like convolution) can be introduced into the temperature image to perform first-order derivative convolution on the temperature image, generating a gradient response map in two directions (x, y), or combining the two-directional gradient response maps into a gradient magnitude map. These two-directional gradient response maps or gradient magnitude maps are used as feature enhancement signals and fused with the temperature image to generate a temperature feature map with enhanced temperature gradient directional information. Specifically, the two-directional gradient response maps and the temperature image are fed into a convolutional layer as three channels. The convolutional layer combines the original temperature intensity with local variation information, or uses the gradient magnitude map to weight the temperature image, enhancing the response in areas with large gradients.

[0149] The feature map output by ResNet and the feature map obtained after gradient convolution are fused, and global average pooling (GAP) is performed on the fused image to generate the melt pool temperature feature vector.

[0150] This step achieves accurate modeling and dynamic perception of the melt pool temperature state. Through ResNet and gradient convolution, it accurately captures the melt pool temperature distribution pattern, thermal diffusion structure and temperature change trend, providing reliable feature support for melt pool state identification and temperature estimation.

[0151] S403 , performing vector fusion on the molten pool texture feature vector and the molten pool temperature feature vector to obtain a second fused feature vector.

[0152] Exemplarily, vector concatenation, weighted fusion, or fusion network mapping may be used to fuse the melt pool texture feature vector with the melt pool temperature feature vector to generate a second fused feature vector.

[0153] S404: Calculate the molten pool offset according to the second fusion feature vector and the welding path, and calculate the average molten pool temperature according to the molten pool temperature feature vector.

[0154] For example, the polarization image or thermal image can be grayscale normalized, high-pass filtered, and binary segmented to obtain a melt pool region mask. Based on the melt pool region mask, the geometric centroid or temperature-weighted centroid of the melt pool region can be calculated to obtain the initial melt pool center coordinates. A lightweight MLP or regression network is used to calculate the offset correction based on the second fused feature vector, and the offset correction is added to the initial melt pool center coordinates to obtain the final melt pool center coordinates.

[0155] The spatial coordinates of the current path point in the welding path are projected into image coordinates through calibration parameters, or the final molten pool center coordinates are back-projected into the welding space, and the difference between the final molten pool center coordinates and the coordinates of the current path point is calculated, which is the molten pool offset.

[0156] The melt pool temperature feature vector can be input into a trained temperature regression model (such as MLP or support vector regression (SVR)), and the model can calculate the average melt pool temperature.

[0157] This step more accurately captures the actual physical center of the molten pool by fusing polarization texture and thermal structural features. The molten pool temperature calculation does not rely on the pixel average of a single temperature image, but is based on deep structural features, which improves robustness and provides high-precision, real-time perceptible key input for closed-loop welding control. It is suitable for welding scenarios with highly reflective, heterogeneous or non-metallic materials.

[0158] In one possible implementation, the laser welding control method further includes:

[0159] S400A,classifies the defect types of the real-time fused image based on the,comprehensive feature vector and calculates the probability of each defect type.

[0160] For example, the comprehensive feature vector can be input into a trained classification model, and the classification model can calculate the probability of each defect type, and the defect type with the highest probability can be selected as the defect existing in the real-time fused image.

[0161] Classification model training process: Fusion images of the real-time welding process are collected, and the comprehensive feature vector of each image is extracted. The corresponding defect type label of the comprehensive feature vector is manually annotated. The comprehensive feature vector of each image and the corresponding defect type are used as a set of samples, and all samples are divided into a training set and a validation set.

[0162] Build a classification model structure, which includes an input layer (with the same dimensions as the comprehensive feature vector), hidden layers (1-2 fully connected layers and ReLU), and an output layer (with a Softmax activation function, outputting the probability of each defect class). Initialize the model weights and load the training set. Each training round involves the following steps: input the comprehensive feature vectors of the training set into the model in batches, output the predicted probability of each defect class, calculate the error between the predicted value and the true label using the cross-entropy loss function, calculate the gradient through backpropagation, and update the parameters using the optimizer. After each round of training, calculate the accuracy and loss on the validation set. If the validation set accuracy no longer improves, terminate the training early and save the optimal model weights (e.g., when the validation accuracy is the highest).

[0163] In another possible implementation, the laser welding control method further includes:

[0164] S400B, classifies the defect types of the real-time polarization image according to the melt pool texture feature vector and calculates the probability of each defect type.

[0165] For example, the melt pool texture feature vector can be input into a trained classification model, which can calculate the probability of each defect type and select the defect type with the highest probability as the defect present in the real-time polarization image. The classification model training method is the same as step S400A.

[0166] S500, determining real-time laser parameters according to the real-time status information of the molten pool, and controlling the laser welding device to weld the workpiece according to the real-time laser parameters.

[0167] For example, a target average melt pool temperature (e.g., 1500°C), an allowable temperature fluctuation range (e.g., ±50°C), an allowable melt pool offset range (e.g., ±0.2 mm), and a compensation rule (laser power and laser speed are adjusted based on the average melt pool temperature) can be pre-set. Assuming an initial laser power P = 800W, an initial laser speed v = 5mm / s, and an initial focus position z = 0mm, and a compensation rule of adjusting the laser power by approximately ±10W for every 10°C temperature difference, if step S400 determines that the average melt pool temperature at the current path point is 1420°C, 80°C lower than the target average melt pool temperature of 1500°C, this indicates insufficient heat input, and the power can be increased by 80°C / 10°C × 10W = 80W. Therefore, the laser power at the next path point is P = 800W + 80W = 880W. If the average melt pool temperature is higher than the target average melt pool temperature (e.g., 1570°C), the laser power at the next path point is adjusted in the opposite direction, decreasing the laser power. Since the temperature response is nonlinear and does not always correspond 1:1 to the laser power, the thermal coupling efficiency between the laser and the welding workpiece is not constant. Rapid and large-scale power adjustments can easily cause disturbances in the molten pool, causing spatter, porosity, burn-through and other problems. Therefore, the single adjustment range of the laser power can be limited to ±150W to avoid welding instability caused by sudden power changes.

[0168] If the average melt pool temperature exceeds the target average melt pool temperature, it indicates excessive weld heat input. Besides reducing the power, the laser speed can be increased; conversely, the laser speed can be reduced. Assuming a compensation rule of ±0.2 mm / s for every 20°C temperature difference, the laser speed is adjusted accordingly. If the average melt pool temperature at the current path point is 1580°C, exceeding the target average melt pool temperature (1500°C) by 80°C, the speed adjustment is 80°C / 20°C × 0.2 = +0.8 mm / s. Therefore, the laser speed at the next path point is v = 5 mm / s + 0.8 mm / s = 5.8 mm / s. Similarly, if the temperature is low, the laser speed at the next path point can be appropriately reduced to increase heat input. The laser speed can be limited to a range of, for example, 3.5 mm / s to 7.0 mm / s to prevent welding instability caused by sudden speed changes.

[0169] If the molten pool offset exceeds the allowable molten pool offset range, it can be determined that the weld spot is misaligned or the material is uneven. The laser path can be adjusted laterally, with the focus adjusted laterally by 0.1mm for every 0.1mm offset. For example, if the offset is +0.25mm, which exceeds the allowable molten pool offset range of ±0.2mm, the laser focus is corrected by +0.25mm, meaning the laser focus at the next path point moves laterally by 0.25mm. If the offset persists and the temperature decreases simultaneously, further consideration can be given to lowering the focal length (z-direction), such as by 0.2mm, to deepen the energy concentration area and improve coupling efficiency.

[0170] The above real-time laser parameters are sent to the motion control module, laser control module and focus control module in the form of control instructions. The control instructions can use industrial communication protocols (such as EtherCAT, CANopen, Modbus, etc.) to ensure fast, synchronous and low-latency transmission. All adjustments can be completed within one control cycle with millisecond-level response capability.

[0171] The motion control module can adjust the movement speed and focus of the laser head according to the speed instruction (the speed of the next path point) and the offset instruction (the laser focus correction amount of the next path point); the laser control module can instantly adjust the energy of the output laser according to the power instruction (the laser power of the next path point), and the laser spot intensity changes rapidly; the focus control module can fine-tune the laser focus depth according to the focal length adjustment instruction to optimize the position of the energy concentration area.

[0172] Through the above steps, the laser welding device can re-collect the real-time status information of the molten pool, judge the changes, generate new laser parameters, and issue new control instructions after welding is completed at each path point, thereby maintaining a dynamic adaptive welding process in the welding cycle.

[0173] Through the above-mentioned dynamic control process, the weld path can be accurately followed even if there are slight geometric errors in the material; fluctuations caused by changes in the thermophysical properties of the workpiece to be welded can be compensated in real time to ensure that the temperature, size and position of the molten pool are always in a stable state, thereby obtaining continuous, stable and high-quality welds.

[0174] In one possible implementation, in step S500, determining real-time laser parameters based on real-time molten pool status information includes:

[0175] S510: Calculate the actual offset of the laser focus based on the molten pool offset, and calculate the laser posture adjustment amount based on the actual offset. The real-time laser parameters include the laser posture adjustment amount and the laser adjustment power.

[0176] For example, since the melt pool offset is a 2D coordinate (image coordinate system) and the laser pose adjustment is a 3D coordinate, the melt pool offset can be mapped to the workpiece or laser head coordinate system using a calibration matrix (camera intrinsic and extrinsic parameters) to obtain the actual laser focus offset. For each direction of the actual offset (x and y), the pose adjustment (laser pose adjustment) in the x and y directions is calculated independently using the PID algorithm.

[0177] This step can eliminate path errors and improve welding quality, and can be extended to real-time compensation and path adaptation of welding trajectories of robot systems.

[0178] S520, calculating the laser adjustment power according to the average temperature of the molten pool, the preset temperature and the current laser power.

[0179] It can be understood that the preset temperature is the optimal average temperature of the molten pool during welding and serves as a reference for laser power control. The preset temperature is determined by the material's physical melting point and a process compensation margin (which can be set between 10 and 30°C to account for laser heat transfer losses, heat dissipation, and workpiece thermal inertia). The preset temperature should be above the material's melting point but not too high to prevent overheating or degradation. For example, if the melting point of PEEK (polyetheretherketone) is 343°C and the process compensation margin is 20°C, the preset temperature for this material is 363°C.

[0180] Material-related thermal response parameters It is the sensitivity coefficient of converting temperature error into power change, which is used to control the response speed and smoothness. The selection of the material is mainly based on its thermal properties, including thermal conductivity, thermal diffusivity and reflectivity. For example, carbon fiber composite materials have high thermal diffusivity and can respond quickly to prevent cooling. It can be 0.2; ceramics have low thermal diffusion and large thermal inertia, so slow adjustment can avoid overshoot. It can be 0.1.

[0181] For example, the average temperature of the molten pool, the preset temperature and the current laser power can be input into the laser adjustment power calculation formula to calculate the laser power (laser adjustment power) of the next path point, that is, , where P(t+1) represents the laser adjustment power, and P(t) represents the current laser power. represents the material-related thermal response parameters, Indicates the preset temperature, Indicates the average temperature of the molten pool. In order to prevent excessive power jumps, resulting in unstable molten pool or burn-through, the maximum adjustment step can be set (such as the maximum allowable power adjustment each time ±50W), that is, If this part exceeds the maximum adjustment step, only the maximum adjustment step is used in the laser adjustment power calculation formula.

[0182] This step dynamically adjusts the laser power through real-time temperature difference feedback, so that the average temperature of the molten pool is always maintained in the ideal process range, avoiding overheating or insufficient heat, and effectively improving the consistency of welding depth, weld formation quality and heat-affected zone control during the welding process.

[0183] In one possible implementation, the laser welding control method further includes:

[0184] S501: If the average temperature of the molten pool is equal to the preset temperature and the probability of each defect type includes a defect type greater than the probability threshold, the laser adjustment power is calculated based on the defect type with a probability greater than the probability threshold and the current laser power.

[0185] Understandably, maintaining the average molten pool temperature at the preset temperature during laser welding does not always guarantee weld quality. Certain defects (such as lack of fusion and porosity) may occur even when the temperature is normal. Therefore, it is possible to proactively fine-tune the laser power based on the defect type to provide quality compensation control.

[0186] For example, since each defect has different causes and responds to different power requirements, a power adjustment factor can be pre-set for each defect. For example, a porosity defect may be caused by an overheated or unstable melt pool, so the power can be reduced, and the power adjustment factor can be set to -80W. A lack of fusion may be caused by insufficient heat input, so the power can be increased, and the power adjustment factor can be set to +100W.

[0187] When the average temperature of the molten pool is equal to the preset temperature, and the probability of each defect type has a defect type greater than the probability threshold, the power adjustment amount can be calculated based on the probability of the defect type, the power adjustment coefficient, and the probability threshold, that is, ,in, Indicates the power adjustment amount, Indicates the power adjustment factor, represents the probability of defect type, Indicates the probability threshold. Add the power adjustment amount to the current laser power to calculate the laser adjustment power.

[0188] Even if the average temperature of the molten pool is normal, this step can actively adjust the laser energy based on quality feedback (defect probability), realizing a complementary closed loop from thermal control to quality-driven control, which helps to suppress the formation of potential defects and improve welding consistency and stability.

[0189] S502, if the average temperature of the molten pool is not equal to the preset temperature, and there is a defect type with a probability greater than the probability threshold in the probability of each defect type, calculate the first adjustment power according to the average temperature of the molten pool, the preset temperature and the current laser power; calculate the laser adjustment power according to the defect type with a probability greater than the probability threshold and the first adjustment power.

[0190] For example, when the average temperature of the molten pool is not equal to the preset temperature, and the probability of each defect type contains a defect type greater than the probability threshold, the first adjustment power can be calculated based on the average temperature of the molten pool, the preset temperature and the current laser power, and the calculation method is the same as step S520.

[0191] The power adjustment coefficient is determined according to the defect type with a probability greater than the probability threshold, and the power adjustment amount is calculated according to the calculation formula of step S501. The power adjustment amount is added to the first adjustment power to obtain the laser adjustment power.

[0192] This step introduces a defect-driven correction mechanism based on temperature control accuracy, which can take into account both thermal balance control and mass compensation control.

[0193] In one possible implementation, the laser welding control method further includes:

[0194] S101: Acquire a first image and a second image. The first image is an image of a checkerboard calibration plate placed on a welding platform captured by a polarization imaging device, and the second image is an image of a checkerboard calibration plate placed on a welding platform captured by a thermal imaging device.

[0195] For example, before welding, a standard checkerboard calibration plate can be placed on the welding platform, ensuring that the plate is stable, flat, and visible. The polarization imaging device is activated to capture an image that completely covers the calibration plate area. This image serves as the first image, recording the calibration plate's imaging results from the polarization camera's perspective. While maintaining the calibration plate's position, the thermal imaging device is activated to capture a thermal image of the same calibration plate. This image serves as the second image, used to characterize the calibration plate's imaging in the thermal imaging system.

[0196] S102: Extract corner positions of the first image and the second image respectively, and calculate internal parameters of the polarization imaging device and the thermal imaging device respectively based on the first image and the second image, wherein the internal parameters include focal length, principal point coordinates, and distortion parameters.

[0197] Exemplarily, the corner detection function in OpenCV can be used to extract the two-dimensional coordinates (image coordinate system) of all corner points of the checkerboard in the first image and the second image respectively, while ensuring that the order of extracted corner points is consistent in the two, to obtain the corner point set of the first image and the corner point set of the second image.

[0198] Based on the actual physical size of the calibration plate (such as a grid size of 10 mm), the three-dimensional world coordinate set of the corner points of the two images can be defined, that is, the actual position of each corner point of the checkerboard in the physical world.

[0199] You can use the calibrateCamera() function of OpenCV to solve the intrinsic parameters of the polarization imaging device (polarization camera) based on the two-dimensional coordinates and three-dimensional world coordinate sets of all corner points in the first image, and obtain the focal length, principal point coordinates, and distortion parameters of the polarization imaging device.

[0200] Similarly, the two-dimensional coordinates and three-dimensional world coordinates of all corner points of the second image are input into the calibrateCamera() function of OpenCV to calculate the focal length, principal point coordinates and distortion parameters of the thermal imaging device.

[0201] S103: Calculate an initial relative rotation matrix and an initial relative translation vector between the polarization imaging device and the thermal imaging device based on the corner point positions and the internal parameters. Determine an initial transformation matrix based on the initial relative rotation matrix and the initial relative translation vector. The initial transformation matrix represents the coordinate mapping relationship between the polarization imaging device and the thermal imaging device.

[0202] For example, the two-dimensional coordinates of all corner points in the first image and the second image, the three-dimensional world coordinate set, and the internal parameters of the polarization imaging device and the thermal imaging device can be input into the cv2.stereoCalibrate() function in OpenCV. The function can solve the initial relative rotation matrix (the rotation of the thermal imaging device coordinate system relative to the polarization imaging device coordinate system) and the initial relative translation vector (the displacement of the thermal imaging device coordinate system relative to the polarization imaging device coordinate system).

[0203] The initial relative rotation matrix and the initial relative translation vector can be combined into a homogeneous transformation matrix (initial transformation matrix) to represent the transformation from thermal imaging device coordinates to polarization imaging device coordinates. The homogeneous transformation matrix can also be transposed to represent the transformation from polarization imaging device coordinates to thermal imaging device coordinates.

[0204] Through these steps, precise geometric alignment and coordinate unification between the two imaging modalities can be achieved, providing a basic guarantee for the subsequent fusion perception, joint feature extraction and spatial registration of polarization images and thermal images, and improving the accuracy and consistency of the welding system.

[0205] In one possible implementation, the laser welding control method further includes:

[0206] During the welding process, it can be assumed that the position and posture of the thermal imaging device remain unchanged, that is, the initial external parameters are constant. Therefore, only the position changes of the polarization imaging device can be tracked, and the coordinate mapping between the thermal imaging device and the polarization imaging device can be updated in real time through the transformation relationship. This avoids the high computational complexity and low accuracy caused by simultaneously positioning both devices in real time.

[0207] S10: Calculate new extrinsic parameters of the polarization imaging device using a PnP algorithm based on the intrinsic parameters of the polarization imaging device and the feature point coordinate set. The extrinsic parameters include a rotation matrix of the polarization imaging device or the thermal imaging device relative to the world coordinate system and a translation vector of the polarization imaging device or the thermal imaging device in the world coordinate system.

[0208] It can be understood that the feature point coordinate set is the two-dimensional coordinates of all corner points in the image captured in real time by the polarization imaging device during the welding process.

[0209] For example, based on the three-dimensional world coordinate set, the internal parameters of the polarization imaging device and the feature point coordinate set, a PnP algorithm (such as the solvePnP() function in OpenCV) can be used to solve the three-dimensional-two-dimensional corresponding points to obtain a new rotation vector and a new translation vector. The new rotation vector can be converted into a new rotation matrix using the Rodriguez formula. The new external parameters of the polarization imaging device include the new rotation matrix and the new translation vector.

[0210] S20 , calculating a real-time relative rotation matrix and a real-time relative translation vector between the polarization imaging device and the thermal imaging device according to the new extrinsic parameters of the polarization imaging device and the initial extrinsic parameters of the thermal imaging device.

[0211] For example, the inverse of the new rotation matrix of the polarization imaging device may be multiplied by the rotation matrix in the initial extrinsic parameters of the thermal imaging device to obtain the real-time relative rotation matrix.

[0212] The real-time relative translation vector can be obtained by subtracting the translation vector in the initial extrinsic parameters of the thermal imaging device from the product of the real-time relative rotation matrix and the new translation vector.

[0213] S30, determining a real-time transformation matrix according to the real-time relative rotation matrix and the real-time relative translation vector.

[0214] For example, the real-time relative rotation matrix and the real-time relative translation vector can be combined into a homogeneous transformation matrix (real-time transformation matrix), which can be used to map points, images, features, etc. in the polarization image coordinate system to the coordinate system of the thermal imaging system in real time.

[0215] By calculating the extrinsic parameters of the polarization imaging device in real time and combining them with the initial extrinsic parameters of the thermal imaging device, the relative position relationship between the two devices is dynamically acquired, thereby achieving the maintenance of geometric consistency of the multimodal imaging system during motion or environmental changes, which is beneficial to the continuous and precise alignment of image fusion, target detection and control instructions based on a unified coordinate system.

[0216] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0217] Corresponding to the laser welding control method described in the above embodiment, the embodiment of the present application further provides a laser welding control device based on machine vision, and each unit of the device can implement each step of the laser welding control method.

[0218] The device includes:

[0219] The acquisition unit is used to acquire an initial polarization image and an initial thermal image of the workpiece to be welded, wherein the workpiece to be welded is a non-metallic material.

[0220] The initial parameter determination unit is used to determine the welding path and initial laser parameters of the workpiece to be welded according to the initial polarization image and the initial thermal imaging image.

[0221] The first control unit is used to control the laser welding device to weld the workpiece to be welded based on the welding path and the initial laser parameters, and simultaneously obtain a real-time polarization image and a real-time thermal imaging image of the workpiece to be welded.

[0222] The molten pool status monitoring unit is used to determine the real-time status information of the molten pool of the workpiece to be welded based on the real-time polarization image, real-time thermal image, and welding path. The real-time molten pool status information includes the molten pool offset and the average molten pool temperature.

[0223] The second control unit is used to determine real-time laser parameters according to the real-time status information of the molten pool, and control the laser welding device to weld the workpiece according to the real-time laser parameters.

[0224] It should be noted that the information interaction, execution process, etc. between the above-mentioned units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0225] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0226] An embodiment of the present application further provides a machine vision-based laser welding control device, comprising a polarization imaging device, a thermal imaging device, a laser welding device, and a control device communicatively connected to the polarization imaging device, the thermal imaging device, and the laser welding device. The control device of the machine vision-based laser welding control device of this embodiment comprises: at least one processor, at least one memory, and a computer program stored in the at least one memory and executable on the at least one processor. When the processor executes the computer program, the machine vision-based laser welding control device implements the steps of any of the aforementioned laser welding control method embodiments, or implements the functions of each unit in the aforementioned device embodiments.

[0227] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the control device of the machine vision-based laser welding control device.

[0228] The control device of the machine vision-based laser welding control device can be a computing device such as a desktop computer, laptop, PDA, or cloud server. The machine vision-based laser welding control device can include, but is not limited to, a processor and memory. It can also include more or fewer components, or a combination of certain components, or different components. For example, it can also include input and output devices, network access devices, buses, etc.

[0229] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0230] In some embodiments, the memory may be an internal storage unit of the control device of the machine vision-based laser welding control device, such as a hard drive or memory of the machine vision-based laser welding control device. In other embodiments, the memory may be an external storage device of the machine vision-based laser welding control device, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory may include both the internal storage unit of the machine vision-based laser welding control device and an external storage device. The memory is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory may also be used to temporarily store data that has been output or is about to be output.

[0231] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0232] An embodiment of the present application provides a computer program product. When the computer program product runs on a machine vision-based laser welding control device, the machine vision-based laser welding control device implements the steps in any of the above-mentioned method embodiments.

[0233] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the machine vision-based laser welding control device, a recording medium, computer memory, read-only memory (ROM), random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. Examples include a USB flash drive, a removable hard drive, a magnetic disk, or an optical disk. In some jurisdictions, based on legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunications signals.

[0234] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0235] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0236] In the embodiments provided in this application, it should be understood that the disclosed machine vision-based laser welding control device, equipment, and method can be implemented in other ways. For example, the above-described machine vision-based laser welding control device and equipment embodiments are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0237] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0238] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A laser welding control method, characterized in that: include: Acquire an initial polarization image and an initial thermal image of a workpiece to be welded; wherein the workpiece to be welded is a non-metallic material; determining a welding path and initial laser parameters of the workpiece to be welded according to the initial polarization image and the initial thermal imaging image; Based on the welding path and the initial laser parameters, controlling the laser welding device to weld the workpiece to be welded, and simultaneously obtaining a real-time polarization image and a real-time thermal image of the workpiece to be welded; Determining real-time status information of the molten pool of the workpiece to be welded based on the real-time polarization image, the real-time thermal image, and the welding path; wherein the real-time status information of the molten pool includes a molten pool offset and an average molten pool temperature; Determining real-time laser parameters according to the real-time state information of the molten pool, and controlling the laser welding device to weld the workpiece to be welded according to the real-time laser parameters; Wherein, determining the welding path and initial laser parameters of the workpiece to be welded according to the initial polarization image and the initial thermal imaging image includes: performing a weighted wavelet transform on the surface polarization degree of the initial polarization image and the baseline temperature gradient of the initial thermal imaging image to generate an initial fused image; Extracting the geometric contour and material anisotropy of the weld region using a single-stream network based on the initial fusion image to obtain a weld geometric feature vector and a material feature vector; wherein the single-stream network is a residual neural network or a convolutional neural network; Locating the starting point and the end point of the weld using a corner detection algorithm according to the initial fused image; registering the weld geometric feature vector, the material feature vector, and the weld start and end points with the three-dimensional point cloud data of the workpiece to be welded, fitting the weld centerline, and generating a welding path for the workpiece to be welded; wherein the three-dimensional point cloud data is obtained by scanning the surface of the workpiece to be welded with a three-dimensional scanning device; The initial laser parameters are determined according to the material characteristic vector.

2. The laser welding control method according to claim 1, wherein: The step of determining the welding path and initial laser parameters of the workpiece to be welded according to the initial polarization image and the initial thermal imaging image comprises: Extracting a texture feature vector using a polarization flow of a first dual-stream network according to the surface polarization degree and surface polarization angle of the initial polarized image; wherein the polarization flow of the first dual-stream network includes a residual neural network and an attention mechanism; Extracting a temperature feature vector using the heat flow of the first two-stream network according to the temperature field of the initial thermal image; wherein the heat flow of the first two-stream network includes a residual neural network and a temporal convolutional network; Performing vector fusion on the texture feature vector and the temperature feature vector to obtain a first fused feature vector; generating a welding path of the workpiece to be welded according to the first fused feature vector and the three-dimensional point cloud data; The initial laser parameters are determined according to the first fused feature vector.

3. The laser welding control method according to claim 1, wherein: Determining the real-time state information of the molten pool of the workpiece to be welded based on the real-time polarization image, the real-time thermal imaging image, and the welding path includes: performing wavelet decomposition on the molten pool polarization degree of the real-time polarization image and the molten pool temperature gradient of the real-time thermal imaging image, and weightedly fusing the low-frequency and high-frequency components of the molten pool polarization degree and the molten pool temperature gradient to generate a real-time fused image; Extracting low-level features and high-level features of the molten pool using the single-stream network according to the real-time fused image to obtain a comprehensive feature vector; The molten pool offset is calculated according to the real-time fusion image and the welding path, and the molten pool average temperature is calculated according to the comprehensive feature vector.

4. The laser welding control method according to claim 3, wherein: Determining the real-time state information of the molten pool of the workpiece to be welded based on the real-time polarization image, the real-time thermal imaging image, and the welding path includes: Extracting a melt pool texture feature vector using a polarization flow of a second dual-stream network according to the melt pool polarization degree of the real-time polarization image; wherein the polarization flow of the second dual-stream network includes a MobileNetV3 and a long short-term memory network; Extracting a molten pool temperature feature vector using the heat flow of the second dual-stream network according to the temperature field of the real-time thermal imaging image; wherein the heat flow of the second dual-stream network includes a residual neural network and gradient convolution; Performing vector fusion on the molten pool texture feature vector and the molten pool temperature feature vector to obtain a second fused feature vector; The molten pool offset is calculated according to the second fusion feature vector and the welding path, and the molten pool average temperature is calculated according to the molten pool temperature feature vector.

5. The laser welding control method according to claim 4, wherein: The method further comprises: classifying the defect types of the real-time fused image according to the comprehensive feature vector and calculating the probability of each defect type; or, The defect types of the real-time polarization image are classified according to the molten pool texture feature vector, and the probability of each defect type is calculated.

6. The laser welding control method according to claim 5, wherein: Determining real-time laser parameters according to the real-time state information of the molten pool includes: According to the molten pool offset, the actual offset of the laser focus is calculated, and the laser posture adjustment amount is calculated according to the actual offset; wherein the real-time laser parameters include the laser posture adjustment amount and the laser adjustment power; The laser adjustment power is calculated according to the average temperature of the molten pool, the preset temperature and the current laser power.

7. The laser welding control method according to claim 6, wherein: The method further comprises: If the average temperature of the molten pool is equal to the preset temperature, and the probability of each defect type includes a defect type greater than a probability threshold, calculating the laser adjustment power according to the defect type with a probability greater than the probability threshold and the current laser power; If the average temperature of the molten pool is not equal to the preset temperature, and there is a defect type with a probability greater than the probability threshold in the probability of each defect type, the first adjustment power is calculated based on the average temperature of the molten pool, the preset temperature and the current laser power; the laser adjustment power is calculated based on the defect type with a probability greater than the probability threshold and the first adjustment power.

8. The laser welding control method according to claim 1, wherein: The method further comprises: Acquire a first image and a second image; wherein the first image is an image of a calibration plate of a checkerboard pattern placed on a welding platform taken by a polarization imaging device, and the second image is an image of a calibration plate of a checkerboard pattern placed on a welding platform taken by a thermal imaging device; Extracting corner point positions of the first image and the second image respectively, and calculating internal parameters of the polarization imaging device and the thermal imaging device respectively based on the first image and the second image; wherein the internal parameters include focal length, principal point coordinates, and distortion parameters; Based on the corner point positions and the internal parameters, an initial relative rotation matrix and an initial relative translation vector between the polarization imaging device and the thermal imaging device are calculated, and an initial transformation matrix is ​​determined based on the initial relative rotation matrix and the initial relative translation vector; wherein the initial transformation matrix is ​​used to characterize the coordinate mapping relationship between the polarization imaging device and the thermal imaging device.

9. The laser welding control method according to claim 8, wherein: The method further comprises: Calculating new extrinsic parameters of the polarization imaging device using a PnP algorithm based on the intrinsic parameters and feature point coordinate set of the polarization imaging device; wherein the extrinsic parameters include a rotation matrix and a translation vector between the polarization imaging device or the thermal imaging device and the world coordinate system; Calculating a real-time relative rotation matrix and a real-time relative translation vector between the polarization imaging device and the thermal imaging device according to the new extrinsic parameters of the polarization imaging device and the initial extrinsic parameters of the thermal imaging device; A real-time transformation matrix is ​​determined according to the real-time relative rotation matrix and the real-time relative translation vector.

Images