Laser welding control method

Abstract

The invention is suitable for the technical field of laser welding, and particularly relates to a laser welding control method which comprises the steps that an initial polarization image and an initial thermal imaging image of a to-be-welded workpiece are obtained; according to the initial polarization image and the initial thermal imaging graph, the welding path and initial laser parameters of the to-be-welded workpiece are determined; based on the welding path and the initial laser parameters, a laser welding device is controlled to weld the to-be-welded workpiece, and meanwhile a real-time polarization image and a real-time thermal image of the to-be-welded workpiece are obtained; according to the real-time polarization image, the real-time thermal image and the welding path, real-time state information of a molten pool of the to-be-welded workpiece is determined; and real-time laser parameters are determined according to the real-time state information of the molten pool, and the laser welding device is controlled to weld the workpiece to be welded according to the real-time laser parameters. According to the method, accurate welding process control is achieved, and the defects of a traditional visual system are overcome.

Description

Technical Field

[0001] This application belongs to the technical field of laser welding, and particularly relates to a laser welding control method. Background Art

[0002] Laser welding technology is widely used in the connection of metal materials. However, with the continuous improvement of the requirements for the welding quality and efficiency of non-metal materials such as thermoplastic plastics in the electronics, automotive, and medical industries, laser welding is gradually applied to the processing of non-metal materials. Through technical means such as transmission laser welding, an efficient, precise, and low heat-affected connection method is achieved. This technology relies on the absorption and transmission characteristics of materials to laser, combined with the development of optics, thermology, and materials science, providing a new path for non-metal welding.

[0003] In the prior art, many non-metal materials, such as plastics, ceramics, glass, etc., have strong surface reflectivity. Traditional vision systems usually rely on reflected light for imaging. However, for highly reflective materials, the reflected light will be too strong, resulting in overexposed images or bright spots, making it difficult to distinguish the actual position and shape of the weld seam.

[0004] It is difficult to clearly display the weld seam of transparent or semi-transparent non-metal materials through visible light reflection, resulting in the traditional vision system being unable to capture the correct welding area.

[0005] Compared with metal materials, many non-metal materials have poor thermal conductivity. Traditional vision systems rely on surface image features to monitor the welding process and cannot directly capture the microscopic changes caused by heat accumulation in the materials during the welding process.

[0006] In summary, in the process of using a traditional vision system to control laser welding of non-metal materials, there is a problem that it is difficult for the traditional vision to capture the weld seam of non-metal materials. Summary of the Invention

[0007] The embodiments of this application provide a laser welding control method, which can solve the problem that in the process of using a traditional vision system to control laser welding of non-metal materials in the related art, it is difficult for the traditional vision to capture the weld seam of non-metal materials.

[0008] In a first aspect, the embodiments of this application provide a laser welding control method, including: Obtain an initial polarization image and an initial thermal image of the workpiece to be welded; wherein, the workpiece to be welded is a non-metal material; Determine the welding path and initial laser parameters of the workpiece to be welded according to the initial polarization image and the initial thermal image; Based on the welding path and the initial laser parameters, control a laser welding device to weld the workpiece to be welded, and simultaneously obtain the real-time polarization image and real-time thermal image of the workpiece to be welded; Determine the real-time state information of the molten pool of the workpiece to be welded according to the real-time polarization image, the real-time thermal imaging map, and the welding path; wherein, the real-time state information of the molten pool includes the molten pool offset and the average temperature of the molten pool. Determine the real-time laser parameters according to the real-time state information of the molten pool, and control the laser welding device to weld the workpiece to be welded according to the real-time laser parameters.

[0009] In the embodiments of the present application, the above technical solutions have at least the following technical effects: The laser welding control method provided by the present application first obtains the initial polarization image and the initial thermal imaging map of the workpiece to be welded (non-metallic material), then determines the welding path and the initial laser parameters of the workpiece to be welded according to the initial polarization image and the initial thermal imaging map, and then controls the laser welding device to weld the workpiece to be welded based on the welding path and the initial laser parameters. At the same time, obtain the real-time polarization image and the real-time thermal imaging map of the workpiece to be welded, and determine the real-time state information of the molten pool of the workpiece to be welded (molten pool offset and average temperature of the molten pool) according to the real-time polarization image, the real-time thermal imaging map, and the welding path. Finally, determine the real-time laser parameters according to the real-time state information of the molten pool, and control the laser welding device to weld the workpiece to be welded according to the real-time laser parameters. This method determines the welding path and laser parameters of the workpiece to be welded through the initial polarization image and the initial thermal imaging map, which is beneficial for the laser welding device to have accurate target positioning and welding condition setting from the very beginning, so as to achieve high-quality welding effects. This method can dynamically monitor the state of the molten pool during the welding process through the real-time polarization image and the thermal imaging map, so that the changes during the welding process can be captured in time, and then adjust the laser parameters according to the real-time feedback to maintain the stability and consistency of the welding. This method can optimize the shape and temperature distribution of the molten pool by continuously monitoring the state of the molten pool and adjusting the laser parameters according to the real-time data, avoiding problems such as overheat treatment or uneven temperature, thereby improving the welding quality and reducing defects and deformations. This method can not only achieve precise control of the welding process, but also make up for the deficiencies of traditional vision systems, provide more accurate weld tracking and real-time feedback, significantly improve the automation and intelligence level of laser welding, and improve the quality and efficiency of non-metallic material welding.

[0010] In a second aspect, an embodiment of the present application provides a laser welding control device based on machine vision, including: An acquisition unit for acquiring an initial polarization image and an initial thermal imaging map of a workpiece to be welded; wherein, the workpiece to be welded is a non-metallic material; An initial parameter determination unit for determining the welding path and the initial laser parameters of the workpiece to be welded according to the initial polarization image and the initial thermal imaging map; A first control unit, configured to control a laser welding device to weld a workpiece to be welded based on the welding path and the initial laser parameters, and simultaneously obtain a real-time polarization image and a real-time thermal image of the workpiece to be welded; A molten pool state monitoring unit, configured to determine real-time state information of the molten pool of the workpiece to be welded according to the real-time polarization image, the real-time thermal image, and the welding path; wherein, the real-time state information of the molten pool includes a molten pool offset and an average temperature of the molten pool; A second control unit, configured to determine real-time laser parameters according to the real-time state information of the molten pool, and control the laser welding device to weld the workpiece to be welded according to the real-time laser parameters.

[0011] In a third aspect, an embodiment of the present application provides a laser welding control device based on machine vision, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method described in any one of the embodiments in the first aspect is implemented.

[0012] It can be understood that the beneficial effects of the above second aspect to the third aspect can be referred to the relevant descriptions in the first aspect above, and will not be repeated here. Description of the Drawings

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0014] Figure 1 is a schematic flowchart of a laser welding control method provided by an embodiment of the present application; Figure 2 is a schematic flowchart of the timing synchronization of data acquisition before welding in the laser welding control method provided by an embodiment of the present application; Figure 3 is a schematic flowchart of the timing synchronization of data acquisition during welding in the laser welding control method provided by an embodiment of the present application. Detailed Embodiments

[0015] In the following description, specific details such as specific system structures and technologies are proposed for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0016] It should be understood that when used in the specification of this application and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their combinations.

[0017] It should also be understood that the term "and / or" as used in the specification of this application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0018] As used in the specification of this application and the appended claims, the term "if" can be construed, depending on the context, as "when", "once", "in response to determining", or "in response to detecting". Similarly, the phrases "if determined" or "if [the described condition or event] is detected" can be construed, depending on the context, to mean "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]".

[0019] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are used only for differentiating descriptions and cannot be construed as indicating or implying relative importance.

[0020] Reference to "one embodiment" or "some embodiments" or the like described in the specification of this application means that a specific feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0021] In the related art, many non-metallic materials, such as plastics, ceramics, glass, etc., have strong surface reflectivity. Traditional vision systems usually rely on reflected light for imaging. The high-brightness regions generated by the reflection of laser beams on the material surface will interfere with the imaging of the camera, resulting in overexposed images or the appearance of bright spots, thus making the welding area unclear and making it difficult to distinguish the actual position and shape of the weld seam.

[0022] For transparent or semi-transparent non-metallic materials (such as glass, transparent plastics, certain composite materials), since these materials have relatively weak absorption of the high energy of the laser, they often exhibit irregular light spots, reflection or gloss regions during welding, and internal reflection or laser penetration may occur in these materials, resulting in the inability of traditional vision systems to capture the correct welding area.

[0023] Many non-metallic materials have a high surface gloss, irregular surface shapes, and are prone to form uneven reflections. When a molten pool is generated during the welding process, surface deformation may obscure the weld edge or produce indistinguishable light spots. Traditional vision systems cannot handle these complex glossy surfaces and irregular surface morphologies well, and are easily interfered by surface irregularities or reflected light, resulting in blurred images or inability to accurately distinguish the weld.

[0024] Compared with metallic materials, many non-metallic materials have poor heat conduction properties, resulting in local heat accumulation during laser welding, and problems such as overheating, cracking or burn-through are likely to occur. Traditional vision technology relies on surface image features to monitor the welding process, but cannot directly capture the microscopic changes in the material due to heat accumulation during welding, especially in the welding process with drastic temperature changes, making it difficult to detect welding defects (such as local overheating, lack of fusion).

[0025] To solve the above problems, the embodiments of the present application provide a laser welding control method. In this method, first, the initial polarization image and the initial thermal imaging map of the workpiece to be welded (non-metallic material) are obtained. Then, based on the initial polarization image and the initial thermal imaging map, the welding path and the initial laser parameters of the workpiece to be welded are determined. Next, based on the welding path and the initial laser parameters, the laser welding device is controlled to weld the workpiece to be welded. At the same time, the real-time polarization image and the real-time thermal imaging map of the workpiece to be welded are obtained, and based on the real-time polarization image, the real-time thermal imaging map and the welding path, the real-time state information of the molten pool of the workpiece to be welded (molten pool offset and average molten pool temperature) is determined. Finally, based on the real-time state information of the molten pool, the real-time laser parameters are determined, and based on the real-time laser parameters, the laser welding device is controlled to weld the workpiece to be welded. This method determines the welding path and laser parameters of the workpiece to be welded through the initial polarization image and the initial thermal imaging map, which is conducive to the laser welding device having accurate target positioning and welding condition setting from the beginning, so as to achieve high-quality welding effects. This method can dynamically monitor the state of the molten pool during the welding process through the real-time polarization image and the thermal imaging map, enabling the changes during the welding process to be captured in a timely manner, and then adjusting the laser parameters according to the real-time feedback to maintain the stability and consistency of the welding. This method can optimize the shape and temperature distribution of the molten pool by continuously monitoring the state of the molten pool and adjusting the laser parameters according to the real-time data, avoiding problems such as overheat treatment or uneven temperature, thereby improving the welding quality and reducing defects and deformations. This method can not only achieve precise control of the welding process, but also make up for the deficiencies of traditional vision systems, provide more accurate weld tracking and real-time feedback, significantly improve the automation and intelligence level of laser welding, and improve the quality and efficiency of non-metallic material welding.

[0026] The laser welding control method provided by the embodiments of the present application can be applied to a laser welding control device based on machine vision. At this time, the laser welding control device based on machine vision is the execution subject of the laser welding control method provided by the embodiments of the present application. The embodiments of the present application do not impose any restrictions on the specific type of the laser welding control device based on machine vision.

[0027] Exemplarily, a laser welding control device based on machine vision may include a polarization imaging device, a thermal imaging device, a laser welding device, and a control device communicatively connected to the polarization imaging device, the thermal imaging device, and the laser welding device. The polarization imaging device is a device capable of capturing the surface polarization image of a non-metallic material and providing information about surface details, texture, smoothness, etc. It may include a multi-spectral polarization camera, a high-resolution polarization imaging camera, etc. The thermal imaging device is a device capable of capturing the thermal image of a non-metallic material and providing information related to the temperature distribution. It may include a high-speed infrared thermal imager, an infrared thermal imaging camera, etc. The laser welding device is a device capable of performing laser welding on a non-metallic material. It may include a fiber laser welding machine, a CO2 laser welding machine, an ultraviolet laser welding machine, etc. The control device is a device capable of controlling the polarization imaging device, the thermal imaging device, and the laser welding device and capable of performing data processing. It may be a tablet computer, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a desktop computer, a computing device or other processing devices connected to a wireless modem, a computer, a laptop computer, etc.

[0028] To better understand the laser welding control method provided by the embodiments of the present application, the following provides an exemplary introduction to the specific implementation process of the laser welding control method provided by the embodiments of the present application.

[0029] Figure 1 The schematic flowchart of the laser welding control method provided by the embodiments of the present application is shown. The laser welding control method includes: S100, obtaining an initial polarization image and an initial thermal image of a workpiece to be welded. Wherein, the workpiece to be welded is a non-metallic material.

[0030] It can be understood that the function of the polarization imaging device is to capture the polarization image of the surface of the workpiece to be welded. It can suppress the specular reflection of non-metallic materials (such as smooth plastics, resin on the surface of carbon fiber) through polarized light, enhance the visualization of surface textures, and detect surface defects such as microcracks and pores. The polarization imaging device can include a multi-spectral polarization camera, a high-resolution polarization imaging camera, etc. Here, a multi-spectral polarization camera can be selected to obtain the polarization image. The multi-spectral polarization camera can switch the polarization directions of 0°, 45°, 90°, and 135°, respectively capture the reflected light at different polarization angles, and provide rich surface information. The active polarization light source can adopt an annular LED array integrated with a linear polarizer, and the light source angle is adjustable to match the surface characteristics of the material (such as the 850nm near-infrared light source penetrates the carbon fiber surface layer).

[0031] The thermal imaging device can capture the thermal image of the workpiece to be welded, can monitor the temperature distribution of the workpiece under low-power laser irradiation, and provide a basis for the dynamic control of temperature changes during the subsequent welding process. The thermal imaging device can include a high-speed infrared thermal imager, an infrared thermal imaging camera, etc. Here, a high-speed infrared thermal imager (such as FLIR X8580SC) can be selected, which has a high frame rate (≥100Hz), high resolution (640×512), and a temperature measurement range covering the welding temperature (0 - 2000°C).

[0032] The function of the 3D scanning device is to obtain the 3D point cloud data of the workpiece to be welded. The 3D scanning device can include a structured light 3D scanner, a laser scanner, etc. Since the surface of non-metallic materials may have different optical characteristics (such as smooth and strong reflectivity), a structured light 3D scanner can be selected here, which is suitable for non-metallic workpieces with high requirements for surface details, can quickly obtain large-area 3D data, and has strong adaptability to materials with strong surface reflectivity.

[0033] Exemplarily, before welding, the laser welding device has not been started, and it is impossible to generate a pulse signal to trigger the simultaneous start of the polarization imaging device, the thermal imaging device, and the 3D scanning device. Then, a PLC can be used to trigger the synchronous start of the three devices. An instruction can be sent to the PLC through the upper computer (i.e., the control device). The PLC is responsible for synchronously starting the three devices, which is conducive to keeping the data acquisition time of all devices consistent and avoiding time misalignment caused by response delay. In order to ensure the time consistency of the acquisition of polarization images, thermal images, and 3D point cloud data, and avoid data misalignment caused by device response delay, a high-precision clock source (such as a GPS synchronous clock) can be used to perform time synchronization on the three devices, and a clock synchronization algorithm can be introduced into the upper computer to ensure that the time stamps of each trigger signal are strictly aligned among the devices.

[0034] The polarization imaging device, the thermal imaging device, and the 3D scanning device are synchronously started through the PLC control signal, such as Figure 2as shown

[0035] The polarization imaging device can irradiate the surface of the workpiece to be welded with a polarized light source. Since there may be different surface roughnesses and structures on the surface of non-metallic materials, the polarization state of the reflected light will be affected. When the multi-spectral polarization camera receives the reflected light, a polarization filter (such as a quarter-wave plate or a polarizer) can be used to separate polarized light in different directions. By changing the angle of the polarization filter, reflected images at different polarization angles can be obtained. The camera can take multiple images at different polarization angles. By processing these images, a polarization image of the surface of the workpiece to be welded can be generated.

[0036] The non-local means (NLM) denoising algorithm can be used to remove the noise in the polarization image, or wavelet threshold denoising can also be applied to remove the noise with higher frequencies. The Stokes vectors (S0, S1, S2) can be extracted from the polarization images (0°, 45°, 90°, 135°). The Stokes vectors can help quantify the information of different polarization states, that is , , , where 、 、 、 are the measured light intensity values corresponding to the polarization angles respectively. The degree of polarization (DoLP) can be calculated according to the Stokes vectors, that is , DoLP describes the polarization intensity of light; the angle of polarization (AoP) can be calculated according to the Stokes vectors, that is , AoP provides the information of the polarization direction of light. A radial and tangential distortion correction model can be used to correct the distortion of the polarization image through the calibration results, and correct the geometric distortion caused by the optical lens in the polarization image to ensure the accuracy of the image. The polarization image captured by the polarization imaging device is preprocessed according to these steps to obtain the initial polarization image.

[0037] To avoid the interference of the ambient temperature, the workpiece to be welded can be placed under stable ambient temperature conditions. The thermal imaging device can compensate for the ambient temperature to eliminate the influence of external factors. At an appropriate distance and angle, the thermal imaging device scans the workpiece to capture the thermal radiation information on the surface of the workpiece. During the scanning process, the processing system inside the thermal imaging device can convert the collected infrared radiation data into a thermal image, and each pixel represents the temperature of a certain point on the surface of the workpiece.

[0038] The thermal imaging device can be calibrated by a blackbody radiation source to establish a mapping table between the gray value and the temperature. The temperature of the blackbody radiation source is known. By measuring the radiation value reflected back to the thermal imaging device, a standard conversion relationship between gray scale and temperature can be established. Non-uniformity correction (NUC) can be used to eliminate the response differences of the pixels of the thermal imaging device (such as different temperature responses of different pixels) to improve the accuracy of the thermal imaging map. NUC calibrates and dynamically compensates the responses of different pixels to ensure the uniformity of the thermal imaging image. According to these steps, the thermal imaging map captured by the thermal imaging device is preprocessed to obtain an initial thermal imaging map, which provides the reference temperature distribution information of the workpiece to be welded before welding.

[0039] The structured light projector of the structured light 3D scanner projects a known light pattern onto the surface of the workpiece to be welded. The light pattern may be a simple stripe, or a more complex grid and dot matrix. The irregular geometry of the workpiece surface causes the light pattern to deform. By taking multiple-angle shots and projecting multiple light patterns, the camera of the scanner captures these deformed patterns in real time. Based on the known light pattern projection and the data captured by the camera, the scanning system can calculate the three-dimensional coordinates of each surface point according to the geometric relationship. All the three-dimensional coordinate points form a complete three-dimensional point cloud, representing the shape and structure of the surface of the workpiece to be welded. Since some noise or inaccurate points may be captured during the scanning process, the point cloud data can be denoised and optimized. If the scanning is carried out from multiple perspectives, the point cloud data from different perspectives can be registered and merged into a unified point cloud later. Using a surface reconstruction algorithm (such as Poisson reconstruction) on the point cloud data, the point cloud data can be converted into a continuous surface model to form a complete three-dimensional model of the workpiece to be welded. Through the structured light 3D scanner, the three-dimensional point cloud data of non-metallic materials can be obtained very efficiently and accurately.

[0040] S200. According to the initial polarization image and the initial thermal imaging map, determine the welding path and the initial laser parameters of the workpiece to be welded.

[0041] Exemplarily, image segmentation algorithms (such as threshold method, GrabCut, etc.) can be used to divide multiple candidate welding areas on the initial polarization image or the initial thermal imaging map, and score each candidate welding area. The scoring factors of the candidate welding areas can include surface flatness, thermal stability, and material consistency. Determine the surface flatness according to the DoLP of the candidate welding area, that is, the higher the DoLP, the more suitable for welding; determine the thermal stability according to the thermal imaging map, that is, the more uniform the temperature distribution, the higher the thermal stability; determine the material consistency according to the AoP of the candidate welding area, that is, too strong material anisotropy is not suitable for welding. Perform a weighted sum of the surface flatness, thermal stability, and material consistency of the candidate welding areas, calculate the score of each candidate welding area, and select the candidate welding area with the highest score as the final welding area.

[0042] If the workpiece to be welded has a preset weld line, the welding path can be extracted through image recognition (such as edge detection, Hough transform); if there is no preset weld, the boundary of the final welding area can be selected and fitted into a straight line / curved welding path. Based on the temperature gradient and reflectivity gradient, the welding path can be smoothed to avoid the welding path passing through high-risk areas, and the connectivity and closure of the welding path can be judged by combining topological analysis.

[0043] The welding path can be decomposed into a series of discrete path points, and each path point represents the specific position of the laser. For each path point, the reflectivity of this point can be calculated according to the reflected light intensity and DoLP at different angles in the initial polarization image; the temperature change rate of this point can be calculated according to the pixel value in the thermal imaging diagram, that is , where represents the temperature change rate of the th path point, represents the temperature change of the th path point (such as the temperature difference between the thermal imaging diagrams of two consecutive frames at the th path point), represents the time interval. The initial laser power of each path point is calculated according to the reflectivity and temperature change rate of each path point, that is , where, represents the initial laser power of the

[0044] th th path point, [[ID=5H]]

[0045] ​​If there is a structured light module or a laser rangefinder, the distance from the surface of the workpiece to be welded at each path point to the laser head can be obtained in real time; if there is only a polarization image, the surface normal of each pixel can be restored according to DoLP and AoLP, and the distance from the surface of each path point to the laser head can be calculated by integration. Calculate the initial laser focal length of each path point according to the distance from the surface of each path point to the laser head, that is , where represents the initial laser focal length of the th path point, represents the initially set reference focal length (such as the focal length when the laser focus falls on the reference height), represents the distance from the surface of each path point to the laser head, represents the reference height.

[0046] Take the initial laser power, initial laser speed, and initial laser focal length of each path point as the initial laser parameters.

[0047] This step calculates the most suitable laser parameters for each welding path point, which is conducive to precise control of each link in the welding process, thereby improving the welding quality, optimizing the process, and reducing manual intervention.

[0048] In a possible implementation, in S200, according to the initial polarization image and the initial thermal imaging map, determine the welding path and initial laser parameters of the workpiece to be welded, including: S210, perform a weighted wavelet transform on the surface polarization degree of the initial polarization image and the baseline temperature gradient of the initial thermal imaging map to generate an initial fusion image.

[0049] Exemplarily, the degree of polarization (DoLP) image (the surface polarization degree exists for each pixel) can be extracted from the initial polarization image, and the method is the same as step S100, which will not be elaborated here. The baseline temperature gradient image can be calculated from the initial thermal imaging map, which can be the absolute value or the direction amplitude map of the spatial temperature gradient.

[0050] The sizes and coordinate systems of the degree of polarization image and the baseline temperature gradient image can be normalized (such as resampling, registration, interpolation), so that the degree of polarization and the temperature gradient are within a unified gray scale range (such as 0-1 or 0-255), which is conducive to numerical comparability during fusion.

[0051] An appropriate wavelet basis can be selected according to the fusion requirements. For example, the Daubechies (db4) wavelet is suitable for preserving the detail intensity, and the Symlet (sym5) wavelet is more suitable for maintaining the edge continuity. The polarization degree image and the baseline temperature gradient image can be respectively decomposed by wavelet for n layers. Each decomposition layer can contain a low-frequency approximation sub-band (A) and three high-frequency detail sub-bands (horizontal H, vertical V, diagonal D). The weights (α and β, and α + β = 1) of the polarization degree image and the baseline temperature gradient image can be set according to the selected wavelet basis. For example, if sym5 is used, α = 0.6 (stronger polarization) and β = 0.4 can be set; if db4 is used, it is the opposite.

[0052] For each decomposition layer, the low-frequency approximation sub-bands and weights of the polarization degree image and the baseline temperature gradient image can be weighted and averaged to obtain the low-frequency approximation sub-band after their fusion. ; The maximum absolute value selection method (highlighting the edge) or the regional energy selection method is used to fuse the three high-frequency detail sub-bands of the two respectively, that is , where is the fused horizontal high-frequency detail sub-band, is the horizontal high-frequency detail sub-band of the polarization degree image, is the horizontal high-frequency detail sub-band of the baseline temperature gradient image. Similarly for (the fused vertical high-frequency detail sub-band), (the fused diagonal high-frequency detail sub-band).

[0053] The fused coefficients , , , are subjected to n-layer inverse wavelet transform to be restored to a complete fused image. Histogram equalization or edge enhancement filtering can be performed on the fused image to enhance the weld structure features, and the background noise or low-response regions can be filtered out to obtain an initial fused image with enhanced geometric edge structure and retained temperature field characteristics.

[0054] This step organically combines polarization and thermal information in the wavelet domain, fully preserves the structural details of the polarization image and the physical properties of the thermal imaging diagram, and realizes deep fusion in a multi-scale manner, greatly improving the accuracy of subsequent recognition and path planning.

[0055] S220. According to the initial fused image, a single-stream network is used to extract the geometric contour and material anisotropy of the weld area to obtain a weld geometric feature vector and a material feature vector. Among them, the single-stream network is a residual neural network or a convolutional neural network.

[0056] It can be understood that a single-stream network refers to a convolutional network structure that only uses one forward transmission path. It can be a convolutional neural network (CNN), such as VGG or MobileNet; it can also be a residual neural network (ResNet), such as ResNet-18 or ResNet-34, which is used to enhance the feature extraction ability and avoid gradient disappearance.

[0057] Exemplarily, the front end of the single-stream network can identify the edges, contours, and texture changes between the weld seam and the background in the initial fusion image, and can extract the geometric boundary features and surface directionality features of the weld seam. The middle end of the single-stream network can strengthen the perception ability of complex structures (discontinuous weld seams and blurred edges), allow skip connections, prevent information loss in the deep network, and perform supplementary learning on hidden features such as weak anisotropic textures and thermal traces in the initial fusion image, and output a multi-channel deep feature map, where each channel corresponds to a specific physical / geometric property (such as direction gradient, heat energy distribution direction, etc.). The back end of the single-stream network can apply global average pooling (GAP) or Flatten + fully connected layers to the deep feature map, mapping the deep feature map into two types of feature vectors, namely, weld seam geometric feature vectors (such as length, width change rate, average curvature, edge density, direction continuity, etc.) and material feature vectors (such as absorptivity, surface roughness, thermal conductivity anisotropy, etc.).

[0058] This step can accurately extract the geometric contour features of the weld seam and the anisotropic features of the material, providing accurate and structured data support for subsequent welding path fitting and laser parameter setting, thereby significantly improving the accuracy of welding path planning and the adaptability of material response.

[0059] S230, based on the initial fusion image, use the corner detection algorithm to locate the start and end points of the weld seam.

[0060] Exemplarily, corner detection aims to identify the turning points or edge endpoints with local strong direction changes in the image. The corner detection algorithm (such as Harris corner detection, FAST corner detection, etc.) can be run on the initial fusion image to obtain a series of corner coordinates (a set of two-dimensional pixel points), and each corner is accompanied by a response intensity value for sorting and screening.

[0061] A contour extraction algorithm (such as contour tracking or skeleton extraction) can be used to fit the weld seam path, sort all the corner points along the weld seam direction, and calculate the two most forward and backward corner points based on the Euclidean distance or contour projection length, which are regarded as the start and end points of the weld seam path.

[0062] If the general direction of the weld seam is known (such as from left to right), the point with the smallest x coordinate can be directly selected as the start point, and the point with the largest x coordinate can be selected as the end point. For vertical or zigzag weld seam paths, the start and end points on the fitted path can be selected.

[0063] The two corner coordinates obtained by the above method represent the starting point and the ending point of the weld seam, and the position accuracy can be visually confirmed in the image. If three-dimensional coordinates are used for subsequent path fitting, the pixel coordinates can be mapped to the three-dimensional coordinates on the surface of the workpiece to be welded, which can be achieved through a calibration matrix or by projecting it into a three-dimensional point cloud coordinate system.

[0064] Through image processing and corner recognition technology, this step realizes the automatic extraction of the key boundary points of the weld seam path, providing spatial anchor points for subsequent welding path planning, trajectory fitting, and initial laser control.

[0065] S240. Register the weld seam geometric feature vector, material feature vector, and the starting point and ending point of the weld seam with the three-dimensional point cloud data of the workpiece to be welded, fit the center line of the weld seam, and generate the welding path of the workpiece to be welded. Among them, the three-dimensional point cloud data is obtained by scanning the surface of the workpiece to be welded with a three-dimensional scanning device.

[0066] Exemplarily, according to the three-dimensional coordinates projected by the starting point and ending point of the weld seam, a general weld seam trend section can be determined, and a local point cloud area containing the weld seam can be cropped out. According to the weld seam geometric feature vector and material feature vector, a set of spatial points (candidate point set) that may form the weld seam path can be screened out in the local point cloud area. The weld seam geometric feature vector is used to constrain the point cloud morphology in the weld seam area and guide the path search direction, and the material feature vector is used to guide the selection of points with specific surface attributes (such as normal change, polarization response area) in the point cloud.

[0067] Sort the candidate point set according to spatial connectivity or direction continuity. A guiding search algorithm based on the starting point - ending point (such as the shortest path algorithm, RRT, or adaptive piecewise fitting) can be used to extract a series of ordered spatial point sets along the weld seam trend. The three-dimensional B-spline curve fitting or Catmull-Rom spline interpolation method can be used to smooth the ordered spatial point set and fit the center line of the weld seam. If the weld seam path involves a three-dimensional surface (such as welding on the surface of a cylinder), the fitted curve can be projected or attached to the point cloud surface.

[0068] The fitted center line of the weld seam can be used as the welding path of the workpiece to be welded, and the sequence of the weld seam center points and their tangential direction, normal direction, and material thermal response direction (for laser head attitude planning) can be output synchronously.

[0069] The welding path generated through this step not only has geometric continuity but also takes into account material characteristics, providing a solid foundation for high-quality laser welding processes.

[0070] S250. Determine the initial laser parameters according to the material feature vector.

[0071] Exemplarily, the material feature vector can be input into a trained multi-layer perceptron (MLP) model. The MLP model can output a set of initial laser parameters (laser speed, laser power, laser focal length / focus). It is determined whether the output result falls within the safe operation range. If it exceeds, it can be reasonably trimmed to facilitate the laser parameters within the set interval range (such as laser power ∈ [600, 1200] W). The final result can be transmitted to the control system for initial welding settings.

[0072] Training process of the MLP model: An actual laser welding test data set (sample data) can be collected. Each group of samples includes a material feature vector and the corresponding initial laser parameters (measured through experiments or set by technicians).

[0073] Constructing the MLP model: 12 neurons can be set in the input layer; 2 - 3 hidden layers can be set, with 64 or 128 neurons in each layer. The activation function can be ReLU; 3 neurons can be set in the output layer (corresponding to power, speed, and focus respectively), and the activation function can be linear or mapped to the real number interval after a limited-range Sigmoid; A Dropout layer or L2 regularization can be added to prevent overfitting.

[0074] The model parameter weights can be initialized. In each iteration, forward propagation, loss calculation, backpropagation, and parameter update can be performed on each batch of sample data. After each iteration, the loss change is evaluated on the validation set. If the validation set loss does not decrease for N consecutive rounds, the early stopping mechanism is triggered, and the model accuracy is evaluated using the test set. The evaluation metrics can include the mean absolute error (MAE), mean relative error (MAPE), and the individual regression accuracies of power, speed, and focus.

[0075] This step can intelligently infer the adapted initial laser parameters from the actual photothermal response characteristics of the material, getting rid of the limitations of manual experience setting and improving the welding quality and automation level.

[0076] In another possible implementation, in S200, according to the initial polarization image and the initial thermal imaging map, the welding path and initial laser parameters of the workpiece to be welded are determined, including: S201, according to the surface degree of polarization and surface polarization angle of the initial polarization image, use the polarization stream of the first two-stream network to extract the texture feature vector. Among them, the polarization stream of the first two-stream network includes a residual neural network and an attention mechanism.

[0077] Exemplarily, a degree of polarization (DoLP) image and a polarization angle (AoP) image can be extracted from the initial polarization image (each pixel has a surface degree of polarization and a surface polarization angle). The two images are used as complementary inputs, respectively reflecting the information of intensity change and direction change. For unified processing, the two can be stacked into a two-channel input tensor.

[0078] It can be understood that the polarization stream is a sub-path in the first dual-stream network, dedicated to extracting surface texture feature vectors from polarization images. The residual neural network can be responsible for extracting low, medium, and high-level features such as the spatial structure, texture density, and edge frequency of the surface texture. The attention mechanism can more effectively identify key texture regions and regions with significant surface direction changes. The attention module can be embedded in the middle and rear parts of the residual structure to enhance the output feature map.

[0079] Exemplarily, a residual neural network (such as ResNet-18) can be used. The dual-channel input tensor is input into ResNet-18 and processed through several residual blocks to extract multi-scale and multi-level spatial texture features, thereby obtaining a polarization texture feature map. Each residual block can include two convolutions, batch normalization, activation (ReLU), and a residual skip connection, with the number of output channels gradually increasing and the spatial resolution gradually decreasing.

[0080] The attention mechanism can be a channel attention mechanism (SE module), which automatically assigns a weight to each feature channel, highlighting direction, edge, and frequency features that are meaningful for texture discrimination, such as emphasizing regions with anisotropy on the surface and weakening flat background regions. It can also be a spatial attention mechanism (CBAM or SAM), which locates texture significant regions in the polarization texture feature map, such as junction lines, fiber intersection points, or corner regions, further enhancing the network's attention ability at key positions and suppressing invalid regions. The channel attention mechanism or the spatial attention mechanism can be used alone, or the channel and spatial attention mechanisms can be used simultaneously to form a hybrid attention mechanism. The output image still maintains the same shape, but important texture information is enhanced in the polarization texture feature map, and irrelevant regions are suppressed.

[0081] After being processed by the residual network and the attention mechanism, the first dual-stream network obtains a set of high-dimensional feature maps. Through the global average pooling (GAP) operation, the high-dimensional feature maps can be compressed into a fixed-length high-dimensional vector (texture feature vector).

[0082] This step realizes the automatic conversion from low-level optical information to high-level material physical information, can be applied to the surfaces of complex materials such as non-metals, can obtain real process characteristics from microscopic directional and texture differences, and provides a quantifiable expression for multi-modal information fusion.

[0083] S202, according to the temperature field of the initial thermal imaging map, use the heat flow of the first dual-stream network to extract the temperature feature vector. Among them, the heat flow of the first dual-stream network includes a residual neural network and a temporal convolutional network.

[0084] Exemplarily, the initial thermal image can be extended to pseudo-temporal data through a sliding window or a simulated time-step construction method, that is, the single-frame thermal image is copied into multiple scale views or its thermal gradient diffusion form is simulated to generate a sequence of multi-scale temperature maps.

[0085] Each frame of the multi-scale temperature map sequence can be input into ResNet-18 to extract spatial gradient features (such as thermal distribution profiles, boundary structures of high and low temperature regions, temperature gradient mutation lines) through multiple residual blocks, generating a temperature feature map for each frame.

[0086] Taking time as the first dimension, the temperature feature maps of all frames can be stacked into a temperature feature map sequence and input into a temporal convolutional network (such as one-dimensional temporal convolution (1D Temporal Conv)). The temporal convolutional network processes the trends of temperature changes with time or scale for each channel along the T-axis, similar to sliding a window in the time dimension, learning the change patterns of temperature responses at each spatial position, and generating a temporal feature map after fusing spatial and thermal diffusion trends. Compared with RNN / LSTM, the temporal convolutional network (TCN) has stronger parallelism and is suitable for spatial-thermal data processing.

[0087] Global average pooling (GAP) or adaptive pooling can be performed on the temporal feature map to form a temperature feature vector that globally represents the thermal behavior of the material.

[0088] This step can effectively extract the thermal response features of the workpiece to be welded under the initial thermal field by combining the extraction of temperature features by the residual neural network and the modeling of thermal diffusion trends by the temporal convolutional network, thus providing deep and physically meaningful temperature feature support for welding path planning and laser parameter setting, and realizing more accurate and adaptive welding control.

[0089] S203, perform vector fusion on the texture feature vector and the temperature feature vector to obtain a first fusion feature vector.

[0090] Exemplarily, vector concatenation, weighted fusion, or fusion network mapping (such as using a small MLP or an attention layer to interactively map the two vectors) can be used to fuse the texture feature vector and the temperature feature vector to generate a single global representation (the first fusion feature vector).

[0091] S204, generate the welding path of the workpiece to be welded according to the first fusion feature vector and the three-dimensional point cloud data.

[0092] Exemplarily, according to the texture and thermal diffusion direction features in the first fusion feature vector, the regions most likely to have welds can be screened out on the surface of the point cloud. For example, if the first fusion feature vector contains a linear strip-shaped thermal concentration trend and a certain direction texture, then continuous strip-shaped regions corresponding in form are searched for in the three-dimensional point cloud to form candidate regions.

[0093] In the candidate region, a series of center point sequences can be extracted according to the surface normal variation, curvature continuity, and direction consistency of the point cloud to represent the central axis trend of the weld. The extraction method can include local normal voting (judging the edge trend) and local curvature minimum path search. The center point sequence can be smoothed using a three-dimensional spline curve fitting or a Catmull-Rom curve to obtain a spatially smooth and continuous weld center line trajectory (welding path). The welding path can be evenly sampled to obtain a sequence of path points, and each path point includes spatial position coordinates, as well as the normal direction and tangent direction at that point.

[0094] This step can achieve the adaptive coupling of the path and material properties, improve welding consistency, and provide high-quality spatial basic data for subsequent laser trajectory control and closed-loop regulation.

[0095] S205. Determine the initial laser parameters according to the first fusion feature vector.

[0096] Exemplarily, an MLP model can also be used. Just replace each group of samples in the sample data of step S250 with the first fusion feature vector and the corresponding initial laser parameters, and the training process is similar.

[0097] Input the first fusion feature vector into the trained MLP model, and the MLP model can output a set of initial laser parameters (laser speed, laser power, laser focal length / focus).

[0098] Through this step, highly adaptable laser parameters can be generated, eliminating the need for manual trial welding, improving the success rate of the first welding, and reducing material loss.

[0099] S300. Based on the welding path and the initial laser parameters, control the laser welding device to weld the workpiece to be welded, and simultaneously obtain the real-time polarization image and real-time thermal imaging of the workpiece to be welded.

[0100] Exemplarily, the welding path and the initial laser power, initial laser speed, and initial laser focal length of each path point can be organized into an internal control data structure (such as a task queue, parameter table, instruction cache). For each path point, a set of control instructions can include a movement instruction for instructing the laser head of the laser welding device to move to , with a speed of ; a laser power setting instruction for setting the output power of the laser welding device to ; a focal length adjustment instruction for setting the focusing mechanism of the laser welding device to adjust to the focal length ; Laser trigger instructions for controlling the laser welding device to emit a laser beam at a set power at the current position. These instructions can be in a standard motion control language (such as G-code) or sent to each control module via a dedicated industrial bus (such as EtherCAT). That is, the movement instructions are sent to the motion control module (for performing path point positioning and speed control), the laser power setting instructions and laser trigger instructions are sent to the laser control module (for setting the laser power and controlling the laser beam on / off), and the focal length adjustment instructions are sent to the focusing control module (for driving the electric lens or lens group to complete focal length adjustment). The controller coordinates all control modules to run synchronously according to the instructions to ensure that the laser irradiates the workpiece at the accurate position with the set parameters.

[0101] During welding, the pulse signal output by the laser can be used to trigger the polarization imaging device and the thermal imaging device for image acquisition, achieving microsecond-level time control, as Figure 3 shown.

[0102] It can be assumed that each welding output pulse of the laser welding device is regarded as the main trigger event, and the image acquisition system (polarization imaging device and thermal imaging device) is the slave system, which executes according to a fixed delay. That is, a TTL trigger signal is sent for each laser pulse, and the TTL trigger signal is sent to the polarization imaging device. After receiving the TTL trigger signal, the polarization imaging device immediately triggers the multi-spectral polarization camera to expose. The exposure time can be set to 100 us, the start time of exposure is t, and the end of exposure is t + 100 us. The real-time polarization image reflects the surface reflection state in a very short time after the laser action; after the polarization exposure is started, a high-precision timer can be used to delay for 200 us, and the high-speed infrared thermal imager is triggered to expose at t + 200 us. The exposure time can be set to 100 us, so the acquisition time is from t + 200 us to t + 300 us. At this time, the thermal imaging captures the early thermal diffusion state after laser heating. Close to the laser pulse, it can capture the change of the reflection surface. Triggering the thermal imaging device with a delay can avoid laser interference with the infrared lens and at the same time make the thermal response form a measurable signal.

[0103] This step realizes the precise control of laser welding based on the welding path and initial laser parameters, and through the timing mechanism synchronized with the laser pulse, it acquires real-time polarization images and thermal imaging images during the welding process with high efficiency and low delay, so as to obtain synchronous visual information of each path point during the welding process, providing high-quality and high-precision basic data support for subsequent molten pool state monitoring and laser parameter adaptive adjustment.

[0104] S400. Determine the real-time state information of the molten pool of the workpiece to be welded according to the real-time polarization image, the real-time thermal imaging image and the welding path. Among them, the real-time state information of the molten pool includes the molten pool offset and the average temperature of the molten pool.

[0105] Exemplarily, for each path point: the pixel values in the real-time thermal imaging map can be converted into actual temperatures through emissivity compensation and the built-in temperature conversion model of the camera (or by looking up a table) (for example, the temperature corresponding to the (50, 120)th pixel is 812.6 °C), and a temperature threshold can be set, and the area greater than the temperature threshold is determined as the first molten pool area.

[0106] The degree of linear polarization (DoLP) at each pixel position can be calculated based on the real-time polarization image, and the calculation method is the same as that in step S100. A DoLP threshold can be set, and the area less than the DoLP threshold is determined as the second molten pool area.

[0107] The area where the first molten pool area and the second molten pool area intersect can be used as the final molten pool area. According to the pixel coordinates of each pixel in the final molten pool area (the real-time polarization image and the real-time thermal imaging map have completed image registration and alignment, and the pixel coordinates in both are the same) and the corresponding temperature values, the coordinates of the center of the molten pool are calculated, that is , , where represents the coordinates of the center of the molten pool, represents the final molten pool area each pixel coordinate in, represents the temperature value of each pixel in the thermal imaging map.

[0108] The pixel coordinates of the center of the weld seam in the image can be calculated according to the welding path, and the offset of the molten pool can be calculated based on the pixel coordinates of the center of the weld seam and the coordinates of the center of the molten pool, that is , where represents the offset of the molten pool, represents the pixel coordinates of the center of the weld seam.

[0109] The average temperature of the molten pool can be calculated according to the number of pixels in the final molten pool area and the temperature value of each pixel, that is , where represents the average temperature of the molten pool, represents the number of pixels in the final molten pool area.

[0110] This step realizes the fusion analysis based on the thermal imaging map, polarization image and welding path, and calculates the offset of the molten pool and the average temperature of the molten pool at each path point in real time, providing a key perception basis for the closed-loop adjustment of subsequent laser parameters, and improving the stability and intelligence of the welding process.

[0111] In a possible implementation, in S400, based on the real-time polarization image, real-time thermal imaging map and welding path, the real-time state information of the molten pool of the workpiece to be welded is determined, including: S410, perform wavelet decomposition on the polarization degree of the molten pool in the real-time polarization image and the temperature gradient of the molten pool in the real-time thermal imaging diagram, and weight and fuse the low-frequency and high-frequency components of the polarization degree of the molten pool and the temperature gradient of the molten pool to generate a real-time fusion image.

[0112] Exemplarily, the method for generating the real-time fusion image is the same as that of step S210 and will not be elaborated here.

[0113] This step can enhance the temperature field characteristics while maintaining the structural details of the molten pool, generating a real-time fusion image with both spatial texture and thermal diffusion information, thereby improving the accuracy and robustness of subsequent molten pool feature extraction and providing more comprehensive and stable visual input for welding state perception and control.

[0114] S420, according to the real-time fusion image, use a single-stream network to extract the low-level features and high-level features of the molten pool to obtain a comprehensive feature vector.

[0115] Exemplarily, the first few layers of the single-stream network (such as small-size convolutional kernels (such as 3×3)) can be used to extract the low-level features of the molten pool (such as detailed features like local edges, molten pool contours, temperature contrast, etc.) to obtain a low-level feature map; the later layers of the single-stream network (multiple layers of convolution and pooling) can be used to extract global features (high-level features) with abstract semantics to obtain a high-level feature map, such as the overall shape and expansion direction of the molten pool, the heat distribution trend at the center of the molten pool, the surface directional change, and the energy concentration area.

[0116] The skip connection and multi-scale splicing methods can be adopted to fuse the low-level feature map and the high-level feature map, perform global average pooling (GAP) on the fused feature map, compress the two-dimensional spatial information into a one-dimensional numerical vector, and obtain a fixed-length comprehensive feature vector (such as 64-dimensional, 128-dimensional, 256-dimensional, etc.).

[0117] This step compresses the complex information in the multi-modal image into a quantifiable and traceable feature vector, enhancing the system's real-time recognition, classification, and adjustment capabilities for the molten pool state.

[0118] S430, according to the real-time fusion image and the welding path, calculate the molten pool offset, and calculate the average temperature of the molten pool according to the comprehensive feature vector.

[0119] It can be understood that the molten pool offset represents the lateral or longitudinal deviation between the actual center of the current molten pool and the ideal welding path, and is used to judge whether the welding deviates from the target trajectory. The average temperature of the molten pool is the key basis for controlling parameters such as laser power and welding speed, and is used to reflect the current welding heat input state.

[0120] Exemplarily, in the real-time fused image, the molten pool area can be extracted by threshold method, edge detection or segmentation algorithm (such as based on heat value or texture intensity) to obtain the molten pool area mask, and according to the molten pool area mask, the centroid of the molten pool area (i.e., the area-weighted center) is calculated, which is the current molten pool center coordinates of the workpiece to be welded.

[0121] The spatial coordinates of the current path point can be determined according to the path point information in the welding path. Since the welding path is in three-dimensional space, the spatial coordinates of the current path point can be projected into image coordinates through calibration parameters, or the current molten pool center coordinates can be back-projected into the welding space to ensure that the current molten pool center coordinates and the coordinates of the current path point are in the same reference system. Calculate the difference between the current molten pool center coordinates and the coordinates of the current path point, which is the molten pool offset.

[0122] Based on the trained regression model (such as MLP, random forest regression, linear regression, etc.), the comprehensive feature vector can be mapped to the average temperature of the molten pool (regression value). If the MLP model is used, each group of samples in the sample data of step S250 can be replaced with the comprehensive feature vector and the corresponding average temperature of the molten pool (which can be measured by a thermal imager or derived from the laser power and material model), and the training process is similar. The molten pool area can also be extracted from the real-time fused image, and the average temperature of the molten pool area is calculated as an estimated value, and the estimated value and the regression value are weighted and summed to improve the stability.

[0123] This step can convert the real-time fused image into a quantifiable physical state quantity of the molten pool, realize the closed-loop bridging from visual perception to welding control, can calculate the offset and temperature of the molten pool with high precision, adapt to different materials and surface states, and provide key real-time feedback parameters for the full-automatic laser welding system.

[0124] In another possible implementation, S400, according to the real-time polarization image, the real-time thermal image and the welding path, determine the real-time state information of the molten pool of the workpiece to be welded, including: S401, according to the polarization degree of the molten pool in the real-time polarization image, use the polarization flow of the second two-stream network to extract the molten pool texture feature vector. Among them, the polarization flow of the second two-stream network includes MobileNetV3 and long short-term memory network.

[0125] It can be understood that MobileNetV3 is a lightweight convolutional neural network with excellent feature expression ability and computing efficiency, suitable for deployment in a real-time welding system. Since the molten pool is a dynamically changing area, it is difficult to accurately identify the molten pool trend only by static frames. The long short-term memory network (LSTM) can capture the temporal correlation in continuous frames and identify the texture change trajectory. By combining MobileNetV3 and LSTM, the spatial texture structure and dynamic change trend of the molten pool during the welding process can be captured.

[0126] Exemplarily, before obtaining the real-time polarization image, the polarization images of the previous T frames are acquired. The degree-of-polarization images of the real-time polarization image and the previous T frames can be extracted. All the degree-of-polarization images are input into MobileNetV3. Through multiple depthwise separable convolutional layers and non-linear activation modules, the network extracts the local and global texture patterns in all the degree-of-polarization images, which may include the molten pool edge structure, internal polarization symmetry, surface perturbation texture and stability, thereby obtaining a sequence of degree-of-polarization feature images.

[0127] Global average pooling (GAP) is performed on each frame image in the sequence of degree-of-polarization feature images to generate a sequence of feature vectors. The sequence of feature vectors is input into an LSTM for temporal modeling. The LSTM can capture the temporal texture changes and output the hidden state vector at the final moment as the molten pool texture feature vector, which synthesizes the spatial structure of the current frame (real-time polarization image) and the evolution in the time dimension.

[0128] S402. According to the temperature field of the real-time thermal imaging, the molten pool temperature feature vector is extracted by using the heat flow of the second two-stream network. The heat flow of the second two-stream network includes a residual neural network and gradient convolution.

[0129] Exemplarily, the original thermal grayscale matrix can be separated from the real-time thermal imaging (or the actual temperature value matrix is obtained through the SDK / API), and a temperature field data matrix is obtained (the value of each pixel point is the temperature or normalized temperature at that position). Essentially, the temperature field is a single-channel image with pixel values representing temperature (temperature image).

[0130] The residual neural network (ResNet) can extract the spatial temperature distribution features of the temperature image through multiple residual blocks, generating a multi-channel temperature spatial feature map. A custom gradient convolution kernel (such as a Sobel-like convolution) can be introduced on the temperature image to perform a first-order derivative convolution on the temperature image, generating gradient response maps in two directions (x, y), or combining the gradient response maps in the two directions into a gradient magnitude map. The gradient response maps in the two directions or the gradient magnitude map are used as feature enhancement signals and fused with the temperature image to generate a temperature feature map that enhances the temperature gradient direction information, that is, the gradient response maps in the two directions and the temperature image are sent as three channels into the convolutional layer, and the convolutional layer combines the original temperature intensity and local change information, or weights the temperature image with the gradient magnitude map so that regions with large gradients receive enhanced responses.

[0131] The feature map output by ResNet and the feature map obtained after gradient convolution are fused, and global average pooling (GAP) is performed on the fused image to generate the molten pool temperature feature vector.

[0132] This step realizes the accurate modeling and dynamic perception of the molten pool temperature state. Through ResNet and gradient convolution, it accurately captures the molten pool temperature distribution pattern, heat diffusion structure, and temperature change trend, providing reliable feature support for molten pool state recognition and temperature estimation.

[0133] S403, fuse the molten pool texture feature vector and the molten pool temperature feature vector to obtain a second fused feature vector.

[0134] Exemplarily, vector splicing, weighted fusion, or fusion network mapping can be used to fuse the molten pool texture feature vector and the molten pool temperature feature vector to generate a second fused feature vector.

[0135] S404, calculate the molten pool offset according to the second fused feature vector and the welding path, and calculate the average molten pool temperature according to the molten pool temperature feature vector.

[0136] Exemplarily, gray normalization, high-pass filtering, and binary segmentation can be performed on the polarization image or thermal imaging map to obtain a molten pool area mask. The geometric centroid or temperature-weighted centroid of the molten pool area can be calculated according to the molten pool area mask to obtain the initial molten pool center coordinates. Use a lightweight MLP or regression network to calculate the offset correction amount based on the second fused feature vector, and add the offset correction amount to the initial molten pool center coordinates to obtain the final molten pool center coordinates.

[0137] Project the spatial coordinates of the current path point in the welding path into image coordinates through calibration parameters, or back-project the final molten pool center coordinates into the welding space, and calculate the difference between the final molten pool center coordinates and the coordinates of the current path point, which is the molten pool offset.

[0138] The molten pool temperature feature vector can be input into a trained temperature regression model (such as MLP or support vector regression (SVR), etc.), and the model can calculate the average molten pool temperature.

[0139] This step more accurately captures the actual physical center of the molten pool by fusing polarization texture and thermal structure features. The molten pool temperature calculation does not depend on the pixel average of a single temperature image, but is based on deep structure features, improving robustness, providing key inputs with high precision and real-time perception for closed-loop welding control, and being applicable to welding scenarios of highly reflective, heterogeneous, or non-metallic materials.

[0140] In a possible implementation manner, the laser welding control method further includes: S400A, classify the defect types of the real-time fusion image according to the comprehensive feature vector, and calculate the probability of each defect type.

[0141] Exemplarily, the comprehensive feature vector can be input into a trained classification model. The classification model can calculate the probability of each defect type, and the defect type with the maximum probability can be selected as the defect existing in the real-time fusion image.

[0142] Classification model training process: Collect the fusion images of the real-time welding process, extract the comprehensive feature vector of each image, and at the same time, manually label the defect type label corresponding to the comprehensive feature vector. Take the comprehensive feature vector of each image and the corresponding defect type as a group of samples, and divide all samples into a training set and a validation set.

[0143] Construct the classification model structure, which can include an input layer (with the same dimension as the comprehensive feature vector), a hidden layer (1 - 2 layers of fully connected and ReLU), and an output layer (Softmax activation function, outputting the probability of each type of defect). Initialize the model weights and load the training set. The following steps are performed in each round of training: Input the comprehensive feature vectors of the training set into the model batch by batch, output the predicted probability of each type of defect, calculate the error between the predicted value and the true label using the cross-entropy loss function, calculate the gradient through backpropagation, and update the parameters using an optimizer. Calculate the accuracy and loss on the validation set after each round of training. If the validation set accuracy no longer improves, terminate early and save the best model weights (such as when the validation accuracy is the highest).

[0144] In another possible implementation manner, the laser welding control method further includes: S400B, classify the defect types of the real-time polarization image according to the molten pool texture feature vector, and calculate the probability of each defect type.

[0145] Exemplarily, the molten pool texture feature vector can be input into a trained classification model. The classification model can calculate the probability of each defect type, and the defect type with the maximum probability can be selected as the defect existing in the real-time polarization image. The classification model training method is the same as that in step S400A.

[0146] S500, determine the real-time laser parameters according to the real-time state information of the molten pool, and control the laser welding device to weld the workpiece to be welded according to the real-time laser parameters.

[0147] Exemplarily, the target average molten pool temperature (such as 1500 °C), the allowable temperature fluctuation range (such as ±50 °C), the allowable molten pool offset range (such as ±0.2 mm), and the compensation rule (adjust the laser power and laser speed according to the average molten pool temperature) can be preset. Assume the initial laser power P = 800 W, the initial laser speed v = 5 mm / s, the initial focus position z = 0 mm, and the compensation rule is that for every 10 °C temperature difference, the laser power is adjusted by approximately ±10 W. If it is known through step S400 that the average molten pool temperature at the current path point is 1420 °C, which is 80 °C lower than the target average molten pool temperature of 1500 °C, it can be shown that the heat input is insufficient, and the power can be increased by 80 °C / 10 °C × 10 W = 80 W. Then the laser power P at the next path point is 800 W + 80 W = 880 W; if the average molten pool temperature is higher than the target average molten pool temperature (such as 1570 °C), the adjustment is reversed, and the laser power at the next path point is reduced. Since the temperature response is non-linear and does not always correspond to the laser power in a 1:1 manner, and the thermal coupling efficiency between the laser and the welded workpiece is not constant, rapid and large adjustments of the power are likely to cause molten pool disturbances, leading to problems such as spatter, pores, and burn-through. Therefore, the single adjustment range of the laser power can be limited within ±150 W to avoid welding instability caused by power mutations.

[0148] If the average molten pool temperature is higher than the target average molten pool temperature, it can be shown that the welding heat input is excessive. In addition to reducing the power, the laser speed can also be increased; conversely, the laser speed is reduced. Assume the compensation rule is that for every 20 °C temperature difference, the laser speed is adjusted by ±0.2 mm / s. The average molten pool temperature at the current path point is 1580 °C, exceeding the target average molten pool temperature (1500 °C) by 80 °C. The speed adjustment is 80 °C / 20 °C × 0.2 = +0.8 mm / s. Then the laser speed at the next path point is v = 5 mm / s + 0.8 mm / s = 5.8 mm / s; similarly, if the temperature is low, the laser speed at the next path point is appropriately reduced to enhance the heat input. The laser speed can be limited in a range such as 3.5 mm / s to 7.0 mm / s to avoid welding instability caused by speed mutations.

[0149] If the molten pool offset exceeds the allowable molten pool offset range, it can be judged that it is caused by the welding light spot deviation or material inhomogeneity. The laser path can be adjusted laterally. For every 0.1 mm of offset, the focus is adjusted laterally by 0.1 mm. For example, the offset is +0.25 mm, exceeding the allowable molten pool offset range of ±0.2 mm, then the laser focus is corrected by +0.25 mm, that is, the entire laser focus at the next path point is laterally shifted by 0.25 mm. If the offset persists and the temperature drops synchronously, the focal length (in the z direction) can be further considered to be lowered, such as lowering the focal length by 0.2 mm to deepen the energy concentration area and improve the coupling efficiency.

[0150] Send the above real-time laser parameters to the motion control module, laser control module, and focusing control module in the form of control instructions. The control instructions can use industrial communication protocols (such as EtherCAT, CANopen, Modbus, etc.) to ensure fast, synchronous, and low-latency transmission, and all adjustments can be completed within one control cycle, with millisecond-level response capabilities.

[0151] The motion control module can adjust the moving speed and focus of the laser head according to the speed instruction (the speed of the next path point) and the offset instruction (the laser focus correction amount of the next path point); the laser control module can instantaneously adjust the energy of the output laser according to the power instruction (the laser power of the next path point), and the intensity of the laser spot changes rapidly; the focusing control module can finely adjust the depth of the laser focus according to the focal length adjustment instruction to optimize the position of the energy concentration area.

[0152] Through the above steps, after welding is completed at each path point, the laser welding device can re-collect the real-time state information of the molten pool, judge the changes, generate new laser parameters, and issue new control instructions to maintain a dynamically adaptive welding process during the welding cycle.

[0153] Through the above dynamic control process, it can accurately follow the weld path even if there are minor geometric errors in the material; it can compensate in real time for the fluctuations caused by changes in the thermal physical properties of the workpiece to be welded, ensuring that the temperature, size, and position of the molten pool are always in a stable state, thereby obtaining continuous, stable, and high-quality welds.

[0154] In a possible implementation, in step S500, according to the real-time state information of the molten pool, determine the real-time laser parameters, including: S510, calculate the actual offset of the laser focus according to the molten pool offset, and calculate the laser pose adjustment amount according to the actual offset. Among them, the real-time laser parameters include the laser pose adjustment amount and the laser adjustment power.

[0155] Exemplarily, since the molten pool offset is a two-dimensional coordinate (image coordinate system), and the laser pose adjustment amount is a three-dimensional coordinate, the molten pool offset can be mapped to the coordinate system of the workpiece to be welded or the laser head through a calibration matrix (camera internal parameters and external parameters) to obtain the actual offset of the laser focus. For each direction (x and y) of the actual offset, the PID algorithm is independently used to calculate the pose adjustment amounts (laser pose adjustment amounts) in the x and y directions.

[0156] This step can eliminate path errors and improve welding quality, and it can be extended to real-time compensation of welding trajectories and path adaptation of robot systems.

[0157] S520, calculate the laser adjustment power according to the average temperature of the molten pool, the preset temperature, and the current laser power.

[0158] It can be understood that the preset temperature is the optimal average temperature value of the molten pool during the welding process and is the target reference for laser power control. The preset temperature can be composed of the physical melting point temperature of the material and the process compensation margin (considering laser heat transfer loss, heat dissipation, workpiece thermal inertia, etc., it can be set to 10~30°C). The preset temperature is higher than the melting point of the material, but should not be too high to avoid overburning or degradation. For example, the melting point of PEEK (polyetheretherketone) material is 343°C, and the process compensation margin is 20°C, then the preset temperature of this material is 363°C.

[0159] Material-related thermal response parameter is the sensitivity coefficient for converting temperature error into power change, and is used to control the response speed and stability. The selection of mainly depends on the thermal physical properties of the material, including thermal conductivity, thermal diffusivity and reflectivity. For example, carbon fiber composite materials have high thermal diffusivity and can respond quickly to prevent cooling, 0 can be 0.2; ceramics have low thermal diffusivity, large thermal inertia, and slow adjustment, which can avoid overshoot, can be 0.1.

[0160] Exemplarily, the average temperature of the molten pool, the preset temperature and the current laser power can be input into the laser adjustment power calculation formula to calculate the laser power (laser adjustment power) of the next path point, that is , where P(t + 1) represents the laser adjustment power, P(t) represents the current laser power, represents the material-related thermal response parameter, represents the preset temperature, represents the average temperature of the molten pool. To prevent excessive power jump and cause instability or burn-through of the molten pool, a maximum adjustment step size (such as a maximum allowable power adjustment of ±50W each time) can be set, that is If this part exceeds the maximum adjustment step size, then only the maximum adjustment step size is used in the laser adjustment power calculation formula.

[0161] This step dynamically adjusts the laser power through real-time temperature difference feedback, so that the average temperature of the molten pool is always maintained in the ideal process range, avoiding overheating or insufficient heat, and effectively improving the penetration consistency, weld forming quality and heat affected zone control effect during the welding process.

[0162] In a possible implementation manner, the laser welding control method further includes: S501, if the average temperature of the molten pool is equal to the preset temperature, and there is a defect type with a probability greater than the probability threshold among the probabilities of each defect type, calculate the laser adjustment power according to the defect type with a probability greater than the probability threshold and the current laser power.

[0163] It can be understood that during the laser welding process, merely maintaining the average temperature of the molten pool equal to the preset temperature does not always guarantee the welding quality, and certain defects (such as lack of fusion, pores, etc.) may occur even when the temperature is normal. Therefore, the laser power can be actively fine-tuned according to the defect type for quality compensation control.

[0164] Exemplarily, since the causes of each type of defect are different and the responses to power are also different, the power adjustment coefficients corresponding to each type of defect can be preset. For example, pore defects may be caused by overheating or instability of the molten pool, and the power can be reduced, so the power adjustment coefficient can be set to -80W; lack of fusion may be caused by insufficient heat input, and the power can be increased, so the power adjustment coefficient can be set to +100W.

[0165] When the average temperature of the molten pool is equal to the preset temperature, and there is a defect type with a probability greater than the probability threshold among the probabilities of each defect type, the power adjustment amount can be calculated according to the probability of this defect type, the power adjustment coefficient, and the probability threshold, that is , where represents the power adjustment amount, represents the power adjustment coefficient, represents the probability of the defect type, represents the probability threshold. Add the power adjustment amount to the current laser power to calculate the adjusted laser power.

[0166] Even when the average temperature of the molten pool is normal, through this step, the laser energy can be actively adjusted based on the quality feedback (defect probability), realizing a supplementary closed-loop from thermal control to quality-driven control, which helps to suppress the formation of potential defects and improve the welding consistency and stability.

[0167] S502. If the average temperature of the molten pool is not equal to the preset temperature, and there is a defect type with a probability greater than the probability threshold among the probabilities of each defect type, calculate the first adjusted power according to the average temperature of the molten pool, the preset temperature, and the current laser power; calculate the adjusted laser power according to the defect type with a probability greater than the probability threshold and the first adjusted power.

[0168] Exemplarily, when there is a situation where the average temperature of the molten pool is not equal to the preset temperature, and there is a defect type with a probability greater than the probability threshold among the probabilities of each defect type, the first adjusted power can be calculated according to the average temperature of the molten pool, the preset temperature, and the current laser power, and the calculation method is the same as that in step S520.

[0169] Determine the power adjustment coefficient according to the defect type with a probability greater than the probability threshold, and calculate the power adjustment amount according to the calculation formula in step S501. Add the power adjustment amount to the first adjusted power to obtain the adjusted laser power.

[0170] This step introduces a defect-driven correction mechanism based on the temperature control accuracy, which can take into account both the thermal balance control and the quality compensation control.

[0171] In a possible implementation, the laser welding control method further includes: S101, obtaining a first image and a second image. Among them, the first image is a calibration plate image of a checkerboard calibration plate placed on the welding platform captured by a polarization imaging device, and the second image is a calibration plate image of the checkerboard calibration plate placed on the welding platform captured by a thermal imaging device.

[0172] Exemplarily, before welding, a standard checkerboard calibration plate can be placed on the welding platform, and it is ensured that the attitude of the calibration plate is stable, flat and visible. Start the polarization imaging device and capture an image that completely covers the calibration plate area as the first image, which records the imaging result of the calibration plate from the perspective of the polarization camera. Keep the position of the calibration plate unchanged, start the thermal imaging device, and capture the thermal image of the same calibration plate as the second image, which is used to characterize the imaging situation of the calibration plate in the thermal imaging system.

[0173] S102, respectively extracting the corner positions of the first image and the second image, and calculating the internal parameters of the polarization imaging device and the thermal imaging device according to the first image and the second image. Among them, the internal parameters include the focal length, the principal point coordinates and the distortion parameters.

[0174] Exemplarily, the corner detection function in OpenCV can be used to extract the two-dimensional coordinates (image coordinate system) of all the corners of the checkerboard in the first image and the second image respectively, and at the same time ensure that the order of the extracted corners of the two is the same, obtaining the corner set of the first image and the corner set of the second image.

[0175] Based on the actual physical size of the calibration plate (such as the grid size is 10mm), the three-dimensional world coordinate sets of the corners of the two images can be defined, that is, the actual positions of each corner of the checkerboard in the physical world.

[0176] The calibrateCamera() function of OpenCV can be used to solve the internal parameters of the polarization imaging device (polarization camera) based on the two-dimensional coordinates of all the corners of the first image and the three-dimensional world coordinate set, obtaining the focal length, the principal point coordinates and the distortion parameters of the polarization imaging device.

[0177] Similarly, input the two-dimensional coordinates of all the corners of the second image and the three-dimensional world coordinate set into the calibrateCamera() function of OpenCV, and solve to obtain the focal length, the principal point coordinates and the distortion parameters of the thermal imaging device.

[0178] S103. Calculate the initial relative rotation matrix and the initial relative translation vector between the polarization imaging device and the thermal imaging device based on the corner point positions and the internal parameters, and determine the initial transformation matrix according to the initial relative rotation matrix and the initial relative translation vector. The initial transformation matrix is used to characterize the coordinate mapping relationship between the polarization imaging device and the thermal imaging device.

[0179] Exemplarily, the two-dimensional coordinates of all corner points in the first image and the second image, the three-dimensional world coordinate set, and the internal parameters of the polarization imaging device and the thermal imaging device can be input into the cv2.stereoCalibrate() function in OpenCV, and the function can calculate the initial relative rotation matrix (the rotation of the thermal imaging device coordinate system relative to the polarization imaging device coordinate system) and the initial relative translation vector (the displacement of the thermal imaging device coordinate system relative to the polarization imaging device coordinate system).

[0180] The initial relative rotation matrix and the initial relative translation vector can be combined into a homogeneous transformation matrix (the initial transformation matrix) to represent the conversion from the thermal imaging device coordinates to the polarization imaging device coordinates. The transpose of the homogeneous transformation matrix can also be taken to represent the conversion from the polarization imaging device coordinates to the thermal imaging device coordinates.

[0181] Through the above steps, precise geometric alignment and coordinate unification between the two imaging modalities can be achieved, providing a basic guarantee for subsequent fusion perception of polarization images and thermal images, joint feature extraction, and spatial registration, and improving the accuracy and consistency of the welding system.

[0182] In a possible implementation, the laser welding control method further includes: During the welding process, it can be assumed that the position and attitude of the thermal imaging device remain unchanged, that is, the initial external parameters are constant. Then, only the position change of the polarization imaging device can be tracked, and the coordinate mapping between the thermal imaging device and the polarization imaging device can be updated in real time through the transformation relationship, avoiding large computational complexity and low accuracy caused by real-time positioning of both devices simultaneously.

[0183] S10. Calculate the new external parameters of the polarization imaging device using the PnP algorithm based on the internal parameters of the polarization imaging device and the set of feature point coordinates. The external parameters include the rotation matrix of the polarization imaging device or the thermal imaging device relative to the world coordinate system and the translation vector of the polarization imaging device or the thermal imaging device in the world coordinate system.

[0184] It can be understood that the set of feature point coordinates is the two-dimensional coordinates of all corner points in the image captured in real time by the polarization imaging device during the welding process.

[0185] Exemplarily, based on a set of three-dimensional world coordinates, the internal parameters of the polarization imaging device, and a set of feature point coordinates, the PnP algorithm (such as the solvePnP() function in OpenCV) can be used to solve for three-dimensional to two-dimensional corresponding points, obtaining a new rotation vector and a new translation vector. The new rotation vector can be converted into a new rotation matrix using the Rodriguez formula. The new external parameters of the polarization imaging device include the new rotation matrix and the new translation vector.

[0186] S20. Calculate the real-time relative rotation matrix and the real-time relative translation vector between the polarization imaging device and the thermal imaging device according to the new external parameters of the polarization imaging device and the initial external parameters of the thermal imaging device.

[0187] Exemplarily, the inverse of the new rotation matrix of the polarization imaging device can be multiplied by the rotation matrix in the initial external parameters of the thermal imaging device to obtain the real-time relative rotation matrix.

[0188] The translation vector in the initial external parameters of the thermal imaging device can be subtracted from the product of the real-time relative rotation matrix and the new translation vector to obtain the real-time relative translation vector.

[0189] S30. Determine the real-time transformation matrix according to the real-time relative rotation matrix and the real-time relative translation vector.

[0190] Exemplarily, the real-time relative rotation matrix and the real-time relative translation vector can be combined into a homogeneous transformation matrix (real-time transformation matrix), which can be used to map points, images, features, etc. in the polarization image coordinate system to the coordinate system of the thermal imaging system in real time.

[0191] By calculating the external parameters of the polarization imaging device in real time and combining with the initial external parameters of the thermal imaging device, the relative pose relationship between the two devices is dynamically obtained, thereby realizing the geometric consistency maintenance of the multi-modal imaging system during movement or environmental changes, which is beneficial to image fusion, target detection, and continuous and accurate alignment of control instructions based on a unified coordinate system.

[0192] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0193] Corresponding to the laser welding control method described in the above embodiments, an embodiment of the present application also provides a laser welding control device based on machine vision. Each unit of the device can implement each step of the laser welding control method.

[0194] The device includes: An acquisition unit, configured to acquire an initial polarization image and an initial thermal image of a workpiece to be welded. The workpiece to be welded is a non-metallic material.

[0195] An initial parameter determination unit for determining a welding path and initial laser parameters of a workpiece to be welded based on an initial polarization image and an initial thermal image.

[0196] A first control unit for controlling a laser welding device to weld a workpiece to be welded based on the welding path and the initial laser parameters, and simultaneously acquiring a real-time polarization image and a real-time thermal image of the workpiece to be welded.

[0197] A molten pool state monitoring unit for determining real-time state information of the molten pool of the workpiece to be welded based on the real-time polarization image, the real-time thermal image, and the welding path. The real-time state information of the molten pool includes a molten pool offset and an average temperature of the molten pool.

[0198] A second control unit for determining real-time laser parameters based on the real-time state information of the molten pool, and controlling the laser welding device to weld the workpiece to be welded according to the real-time laser parameters.

[0199] It should be noted that the information interaction, execution process, etc. between the above units, due to being based on the same concept as the method embodiment of the present application, for their specific functions and the technical effects brought, reference can be specifically made to the method embodiment part, and details will not be repeated here.

[0200] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above division of each functional unit and module is used as an example. In practical applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working process of the units and modules in the above device can refer to the corresponding process in the foregoing method embodiment, and details will not be repeated here.

[0201] The embodiment of the present application further provides a laser welding control device based on machine vision. The laser welding control device based on machine vision includes a polarization imaging device, a thermal imaging device, a laser welding device, and a control device communicatively connected to the polarization imaging device, the thermal imaging device, and the laser welding device. The control device of the laser welding control device based on machine vision in this embodiment includes: at least one processor, at least one memory, and a computer program stored in the at least one memory and executable on the at least one processor. When the processor executes the computer program, the laser welding control device based on machine vision implements the steps in any of the above-mentioned embodiments of the laser welding control methods, or the functions of each unit in the above-mentioned device embodiments.

[0202] Exemplarily, the computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the control device of the laser welding control device based on machine vision.

[0203] The control device of the laser welding control device based on machine vision may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The laser welding control device based on machine vision may include, but is not limited to, a processor and a memory. It may include more or fewer components, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, a bus, etc.

[0204] The processor may be a Central Processing Unit (CPU), and the processor may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0205] The memory may be an internal storage unit of the control device of the machine vision-based laser welding control device in some embodiments, such as the hard disk or memory of the machine vision-based laser welding control device. The memory may also be an external storage device of the machine vision-based laser welding control device in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the machine vision-based laser welding control device. Further, the memory may also include both the internal storage unit and the external storage device of the machine vision-based laser welding control device. The memory is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory may also be used to temporarily store data that has been output or will be output.

[0206] An embodiment of the present application also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0207] An embodiment of the present application provides a computer program product, and when the computer program product runs on a machine vision-based laser welding control device, the machine vision-based laser welding control device implements the steps in any of the above method embodiments.

[0208] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of this application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the laser welding control device based on machine vision, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, USB flash drive, mobile hard disk, magnetic disk or optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0209] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0210] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed in this article can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0211] In the embodiments provided in this application, it should be understood that the disclosed laser welding control device, equipment and method based on machine vision can be implemented in other ways. For example, the laser welding control device and equipment embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0212] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A laser welding control method, characterized in that, Including: Obtain the initial polarization image and the initial thermal image of the workpiece to be welded; wherein, the workpiece to be welded is a non-metallic material; Determine the welding path and the initial laser parameters of the workpiece to be welded according to the initial polarization image and the initial thermal image; Based on the welding path and the initial laser parameters, control the laser welding device to weld the workpiece to be welded, and simultaneously obtain the real-time polarization image and the real-time thermal image of the workpiece to be welded; Determine the real-time state information of the molten pool of the workpiece to be welded according to the real-time polarization image, the real-time thermal image and the welding path; wherein, the real-time state information of the molten pool includes the molten pool offset and the average temperature of the molten pool; Determine the real-time laser parameters according to the real-time state information of the molten pool, and control the laser welding device to weld the workpiece to be welded according to the real-time laser parameters.

2. The laser welding control method according to claim 1, wherein The determining the welding path and the initial laser parameters of the workpiece to be welded according to the initial polarization image and the initial thermal image includes: Perform weighted wavelet transform on the surface polarization degree of the initial polarization image and the baseline temperature gradient of the initial thermal image to generate an initial fusion image; According to the initial fusion image, use a single-stream network to extract the geometric contour and material anisotropy of the weld area to obtain a weld geometric feature vector and a material feature vector; wherein, the single-stream network is a residual neural network or a convolutional neural network; Locate the starting point and the ending point of the weld according to the initial fusion image by using a corner detection algorithm; Register the weld geometric feature vector, the material feature vector, and the starting point and the ending point of the weld with the three-dimensional point cloud data of the workpiece to be welded, fit the weld center line, and generate the welding path of the workpiece to be welded; wherein, the three-dimensional point cloud data is obtained by scanning the surface of the workpiece to be welded by a three-dimensional scanning device; Determine the initial laser parameters according to the material feature vector.

3. The laser welding control method according to claim 2, characterized in that, The determining the welding path and the initial laser parameters of the workpiece to be welded according to the initial polarization image and the initial thermal image includes: Extract a texture feature vector by using the polarization stream of the first two-stream network according to the surface polarization degree and the surface polarization angle of the initial polarization image; wherein, the polarization stream of the first two-stream network includes a residual neural network and an attention mechanism; Extract a temperature feature vector by using the heat stream of the first two-stream network according to the temperature field of the initial thermal image; wherein, the heat stream of the first two-stream network includes a residual neural network and a temporal convolutional network; Perform vector fusion on the texture feature vector and the temperature feature vector to obtain a first fusion feature vector; Generate the welding path of the workpiece to be welded according to the first fusion feature vector and the three-dimensional point cloud data; Determine the initial laser parameters according to the first fusion feature vector.

4. The laser welding control method according to claim 2, wherein: The determining the real-time state information of the molten pool of the workpiece to be welded according to the real-time polarization image, the real-time thermal image and the welding path includes: Perform wavelet decomposition on the polarization degree of the molten pool in the real-time polarization image and the temperature gradient of the molten pool in the real-time thermal imaging map, and perform weighted fusion on the low-frequency and high-frequency components of the polarization degree of the molten pool and the temperature gradient of the molten pool to generate a real-time fusion image; According to the real-time fusion image, use the single-stream network to extract the low-level features and high-level features of the molten pool to obtain a comprehensive feature vector; According to the real-time fusion image and the welding path, calculate the offset of the molten pool, and calculate the average temperature of the molten pool according to the comprehensive feature vector.

5. The laser welding control method according to claim 4, wherein, Based on the real-time polarization image, the real-time thermal imaging map and the welding path, determine the real-time state information of the molten pool of the workpiece to be welded, including: According to the polarization degree of the molten pool in the real-time polarization image, use the polarization stream of the second two-stream network to extract the texture feature vector of the molten pool; wherein, the polarization stream of the second two-stream network includes MobileNetV3 and long short-term memory network; According to the temperature field of the real-time thermal imaging map, use the heat stream of the second two-stream network to extract the temperature feature vector of the molten pool; wherein, the heat stream of the second two-stream network includes a residual neural network and gradient convolution; Perform vector fusion on the texture feature vector of the molten pool and the temperature feature vector of the molten pool to obtain a second fusion feature vector; According to the second fusion feature vector and the welding path, calculate the offset of the molten pool, and calculate the average temperature of the molten pool according to the temperature feature vector of the molten pool.

6. The laser welding control method according to claim 5, characterized in that The method further includes: According to the comprehensive feature vector, classify the defect types of the real-time fusion image and calculate the probability of each defect type; Or, According to the texture feature vector of the molten pool, classify the defect types of the real-time polarization image and calculate the probability of each defect type.

7. The laser welding control method according to claim 6, characterized in that Based on the real-time state information of the molten pool, determine the real-time laser parameters, including: According to the offset of the molten pool, calculate the actual offset of the laser focus, and calculate the laser pose adjustment amount according to the actual offset; wherein, the real-time laser parameters include the laser pose adjustment amount and the laser adjustment power; According to the average temperature of the molten pool, the preset temperature and the current laser power, calculate the laser adjustment power.

8. The laser welding control method according to claim 7, wherein The method further includes: If the average temperature of the molten pool is equal to the preset temperature, and there is a defect type with a probability greater than the probability threshold among the probabilities of each defect type, calculate the laser adjustment power according to the defect type with a probability greater than the probability threshold and the current laser power; If the average temperature of the molten pool is not equal to the preset temperature, and there is a defect type with a probability greater than the probability threshold among the probabilities of each defect type, calculate the first adjustment power according to the average temperature of the molten pool, the preset temperature and the current laser power; calculate the laser adjustment power according to the defect type with a probability greater than the probability threshold and the first adjustment power.

9. The laser welding control method according to claim 1, wherein, The method further includes: Obtain a first image and a second image; wherein, the first image is a calibration plate image of a checkerboard calibration plate placed on a welding platform captured by a polarization imaging device, and the second image is a calibration plate image of a checkerboard calibration plate placed on a welding platform captured by a thermal imaging device; Extract the corner positions of the first image and the second image respectively, and calculate the internal parameters of the polarization imaging device and the thermal imaging device according to the first image and the second image; wherein, the internal parameters include focal length, principal point coordinates and distortion parameters; Calculate the initial relative rotation matrix and the initial relative translation vector between the polarization imaging device and the thermal imaging device according to the corner positions and the internal parameters, and determine the initial transformation matrix according to the initial relative rotation matrix and the initial relative translation vector; wherein, the initial transformation matrix is used to characterize the coordinate mapping relationship between the polarization imaging device and the thermal imaging device.

10. The laser welding control method according to claim 9, wherein The method further includes: Calculate the new external parameters of the polarization imaging device by using the PnP algorithm according to the internal parameters of the polarization imaging device and the set of feature point coordinates; wherein, the external parameters include the rotation matrix and the translation vector between the polarization imaging device or the thermal imaging device and the world coordinate; Calculate the real-time relative rotation matrix and the real-time relative translation vector between the polarization imaging device and the thermal imaging device according to the new external parameters of the polarization imaging device and the initial external parameters of the thermal imaging device; Determine the real-time transformation matrix according to the real-time relative rotation matrix and the real-time relative translation vector.

Images