Laser welding defect identification method and device, electronic equipment and storage medium

Abstract

The application provides a laser welding defect recognition method and device, electronic equipment and storage medium. The method comprises: continuously collecting images of a welding area within a preset time length to obtain a time sequence image stream; performing sliding window translation on the time sequence image stream to obtain a plurality of windows; for each frame of image in a window, performing geometric feature extraction on a keyhole area to obtain geometric features of the keyhole corresponding to the keyhole area; performing photometric feature extraction on a plasma plume area to obtain photometric features of the plasma plume corresponding to the plasma plume area; performing dynamic feature extraction on the geometric features and the photometric features corresponding to N frames of images in a window respectively to obtain a space-time feature vector representing dynamic behavior of the images in the window; and inputting the space-time feature vector corresponding to each window into a preset time sequence deep learning model to obtain a defect recognition result of the welding area. The application can improve the accuracy of laser welding defect recognition.

Description

Technical Field

[0001] This application relates to the field of laser advanced manufacturing technology, and in particular to a laser welding defect identification method, device, electronic device and medium. Background Technology

[0002] Laser welding technology, especially keyhole welding, has been widely used in high-end manufacturing industries such as automotive, aerospace, and power batteries due to its advantages such as high energy density, fast welding speed, small heat-affected zone, and minimal deformation. However, laser keyhole welding is a complex nonlinear process involving the coupling of multiple physical fields, including laser-matter interaction, molten pool flow, and keyhole dynamics. The dynamic instability of the keyhole, such as violent oscillations, collapse, and wobble, is the root cause of welding defects (such as porosity, spatter, burn-through, lack of fusion, and collapse). These defects severely affect the mechanical properties and service reliability of the welded joint. Therefore, achieving online real-time monitoring and intelligent defect identification of the laser welding process is crucial for ensuring welding quality and improving production efficiency.

[0003] Existing laser welding monitoring technologies are mainly divided into indirect monitoring based on acoustic, optical, and electrical signals, and direct monitoring based on vision. Indirect methods such as acoustic sensing, spectral analysis, or photodiodes can capture signals related to process stability, but these signals have weak correlation with specific defect types, low signal-to-noise ratios, and are easily affected by environmental interference. Visual monitoring methods based on high-speed cameras can directly observe the morphology and behavior of the keyhole, molten pool, and plasma, providing the richest and most intuitive process information, and are considered the most promising monitoring methods currently available.

[0004] In the field of visual monitoring, early studies mostly employed off-axis or parallel-axis observation, which was easily obstructed by plasma and metal vapor, making it difficult to clearly capture the dynamic behavior of the core area—the keyhole. Coaxial observation technology, by placing a beam-splitting element inside the laser head, makes the camera's field of view coaxial with the laser beam, enabling it to penetrate plasma and clearly and stably observe the true morphology of the keyhole opening, and is currently the mainstream approach. However, how to efficiently and accurately extract key information closely related to the defect formation mechanism from the acquired high-speed video stream and establish a reliable defect identification model remains a core challenge in this field.

[0005] Therefore, a new method for identifying defects in laser welding is urgently needed to solve the above problems. Summary of the Invention

[0006] In view of this, this application provides a laser welding defect identification method, apparatus, electronic device, and storage medium, which can improve the accuracy of laser welding defect identification.

[0007] A first aspect of this application provides a laser welding defect identification method, comprising: continuously acquiring images of the welding area within a preset time period during laser welding to obtain a time-series image stream; performing a sliding window translation on the time-series image stream to obtain multiple windows, wherein each window includes N frames of the images, where N is an integer greater than 1; for each frame of the images within a window, determining a keyhole region and a plasma plume region in the image; extracting geometric features from the keyhole region to obtain the geometric features of the keyhole corresponding to the keyhole region; extracting photometric features from the plasma plume region to obtain the photometric features of the plasma plume corresponding to the plasma plume region; extracting dynamic features from the geometric features and photometric features corresponding to the N frames of the images within a window to obtain a spatiotemporal feature vector characterizing the dynamic behavior of the images within the window; and inputting the spatiotemporal feature vector corresponding to each window into a preset temporal deep learning model to obtain a defect identification result for the welding area.

[0008] In one possible implementation, the step of dynamically extracting the geometric features and photometric features corresponding to the N frames of images within a window to obtain a spatiotemporal feature vector characterizing the dynamic behavior of the images within the window includes: calculating the mean, standard deviation, and spectrum of the geometric features or photometric features corresponding to the N frames of images within a window according to the following formula: ; ; ;in, The mean is... The standard deviation is... For the spectrum, For any one of the geometric features or the photometric features, Extract the dominant frequency with the highest power spectral density from the spectrum; construct the spatiotemporal feature vector based on the mean, standard deviation and dominant frequency corresponding to each geometric feature or photometric feature.

[0009] In one possible implementation, the geometric features include the keyhole area, the equivalent diameter of the keyhole, the centroid position of the keyhole, and the aspect ratio of the keyhole. Extracting the geometric features of the keyhole region to obtain the geometric features of the keyhole corresponding to the keyhole region includes: obtaining the total number of first pixels in the keyhole region and using the total number of first pixels as the keyhole area; calculating the equivalent diameter of the keyhole based on the keyhole area; obtaining the centroid position of the keyhole in the keyhole region; and calculating the length-to-width ratio of the minimum bounding rectangle of the keyhole region to obtain the aspect ratio of the keyhole.

[0010] In one possible implementation, the photometric features include the plasma plume area, the average plasma gray value, the plasma gray standard deviation, and the number of splashes per unit time. The step of extracting photometric features from the plasma plume region to obtain the photometric features of the plasma plume corresponding to the plasma plume region includes: obtaining the total number of second pixels in the plasma plume region and using the total number of second pixels as the plasma plume area; calculating the average brightness of all pixels in the plasma plume region to obtain the average plasma gray value; calculating the plasma gray standard deviation based on the average brightness; and calculating the number of splashes per unit time based on a splash detection algorithm.

[0011] In one possible implementation, the step of inputting the spatiotemporal feature vector corresponding to each window into a preset temporal deep learning model to obtain the defect identification result of the welding area includes: constructing a spatiotemporal feature vector sequence based on the spatiotemporal feature vector corresponding to each window; inputting the spatiotemporal feature vector sequence into the temporal deep learning model to obtain a probability distribution vector, wherein the probability distribution vector represents the probability of different defect categories existing in the welding area.

[0012] In one possible implementation, the temporal deep learning model includes an input layer, a one-dimensional convolutional layer, a long short-term memory (LSM) network layer, a fully connected layer, and an output layer. The temporal deep learning model outputs the probability distribution vector in the following manner: the input layer receives the spatiotemporal feature vector sequence; the one-dimensional convolutional layer slides along the time dimension, automatically learning the local correlation patterns between elements within the spatiotemporal feature vector sequence to obtain a target feature vector sequence; the LSM network layer receives the target feature vector sequence, captures the long-term dependencies in the target feature vector sequence, and learns the evolutionary pattern of the welding process from the past to the present; the fully connected layer integrates information from the final hidden state of the LSM network layer and outputs the probability distribution vector through the output layer.

[0013] In one possible implementation, continuously acquiring images of the welding area within a preset time period during the laser welding process includes: continuously acquiring the images of the welding area within the preset time period at a preset frame rate and a preset resolution during the laser welding process.

[0014] Secondly, this application also provides a laser welding defect identification device, including: an acquisition module, a sliding window module, a determination module, a geometric feature extraction module, a photometric feature extraction module, a dynamic feature extraction module, and an input module; the acquisition module is used to continuously acquire images of the welding area within a preset time period during the laser welding process to obtain a time-series image stream; the sliding window module is used to perform sliding window translation on the time-series image stream to obtain multiple windows, wherein each window includes N frames of the images, where N is an integer greater than 1; the determination module is used to determine the keyhole region and the plasma plume region in each frame of the image within a window; The geometric feature extraction module is used to extract geometric features from the keyhole region to obtain the geometric features of the keyhole corresponding to the keyhole region; the photometric feature extraction module is used to extract photometric features from the plasma plume region to obtain the photometric features of the plasma plume corresponding to the plasma plume region; the dynamic feature extraction module is used to extract dynamic features from the geometric features and photometric features corresponding to N frames of the image within a window to obtain a spatiotemporal feature vector characterizing the dynamic behavior of the image within the window; the input module is used to input the spatiotemporal feature vector corresponding to each window into a preset temporal deep learning model to obtain the defect identification result of the welding area.

[0015] Thirdly, embodiments of this application also provide an electronic device, the electronic device including a processor and a memory, the memory being used to store instructions, and the processor being used to call the instructions in the memory, causing the electronic device to execute the laser welding defect identification method as described in the first aspect.

[0016] Fourthly, embodiments of this application also provide a computer-readable storage medium that stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the laser welding defect identification method as described in the first aspect.

[0017] Compared with related technologies, the embodiments of this application have at least the following advantages: By continuously acquiring images of the welding area within a preset time period, a time-series image stream is obtained. Then, the time-series image stream is shifted through a sliding window to obtain multiple windows, each containing N consecutive frames of welding area images. For each frame of the same window, geometric features of the keyhole region and photometric features of the plasma plume region are extracted. Then, dynamic features are extracted from all geometric and photometric features of the same window, thereby constructing a high-dimensional spatiotemporal feature vector that integrates geometric, photometric, and dynamic features. This spatiotemporal feature vector provides much richer process information than a single feature or indirect physical quantity (such as absorptivity). This multimodal information fusion greatly enhances the ability of the time-series deep learning model to represent complex welding processes, enabling the model to more accurately distinguish between normal states and various defect states. Especially for internal defects with complex formation mechanisms, such as porosity and lack of fusion, the recognition accuracy is significantly improved, thus enhancing the accuracy of laser welding defect identification. Furthermore, by capturing the keyhole and plasma dynamics behavior, unstable signs in the early stages of defect formation can be identified. For example, before burn-through occurs, the model can detect the continuous expansion and drastic fluctuations of the keyhole area; in areas with dense porosity formation, abnormal changes in the keyhole oscillation frequency can be observed. This enables the laser welding defect identification method of this application not only to perform post-processing identification but also to provide early warning, gaining valuable time for intervention. The entire identification process can achieve millisecond-level response on a high-performance computing platform, meeting the real-time requirements of industrial production.

[0018] The technical effects achieved by the second, third, and fourth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0019] Figure 1 A flowchart illustrating the steps of a laser welding defect identification method provided in an embodiment of this application; Figure 2 Another flowchart of the laser welding defect identification method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an online identification system provided in an embodiment of this application; Figure 4 A diagram of a CNN-LSTM hybrid deep learning model architecture provided in an embodiment of this application; Figure 5 A functional block diagram of a laser welding defect identification device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0021] The following description sets forth many specific details to provide a full understanding of this application. The described embodiments are only some, not all, of the embodiments of this application.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0023] It should be further noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0024] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0025] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0026] For ease of understanding, some concepts related to the embodiments of this application are illustrated and explained by way of example for reference.

[0027] CNN-LSTM is a deep learning model that combines Convolutional Neural Networks (CNN) and Long Short-Term Memory Networks (LSTM), primarily used to process sequence data with spatiotemporal characteristics. Its core advantage lies in combining the spatial feature extraction capabilities of CNNs with the time series modeling capabilities of LSTMs, and it is widely used in time series forecasting, video analysis, weather forecasting, and other fields.

[0028] Keyhole: This is the core physical phenomenon in laser deep penetration welding (also known as keyhole welding). When the laser power density exceeds 10^6 W / cm², the material surface forms high-pressure steam due to violent vaporization. The back pressure overcomes the surface tension and gravity of the liquid metal, displacing the molten metal to form deep and narrow steam holes.

[0029] Plasma plume: During laser welding, a high-power laser beam acts on the surface of the material, causing the temperature in a localized area to rise rapidly to a melting or even vaporizing state. At this time, the metal vapor formed by the material evaporation ionizes with gas molecules such as nitrogen and oxygen in the air, forming a plasma plume composed of high-temperature ionized particles.

[0030] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of an embodiment of the laser welding defect identification method of this application. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0031] It should be noted that the laser welding defect identification method of this application embodiment can be applied to laser welding scenarios, and its execution subject can be a laser welding defect identification device. For example, in a laser welding scenario, a laser welding defect identification device can be used to identify laser welding defects. Of course, the laser welding defect identification method of this application embodiment can also be applied to other scenarios that require laser welding defect identification, and this application does not specifically limit it in this regard.

[0032] The specific process of this embodiment is as follows: Figure 1 As shown, it includes the following steps: S101, during the laser welding process, continuously acquire images of the welding area within a preset time period to obtain a time-series image stream.

[0033] In some embodiments, images of the welding area are continuously acquired within a preset time period at a preset frame rate and a preset resolution.

[0034] It is understood that this embodiment does not specifically limit the size of the preset frame rate, preset resolution, and preset duration, and can be set according to actual needs.

[0035] S102, perform a sliding window translation on the time series image stream to obtain multiple windows, where each window includes N frames of images, where N is an integer greater than 1.

[0036] S103, for each frame of an image within a window, determines the keyhole region and plasma plume region in the image.

[0037] In some embodiments, for each frame image The keyhole region in the image is accurately segmented using image segmentation algorithms (such as Mask R-CNN, a deep learning-based instance segmentation network, or U-Net, a semantic segmentation network).

[0038] In some embodiments, the brightness, area, and morphology of the plasma / metal vapor plume associated with the keyhole also reflect the energy coupling and keyhole stability. The plasma plume region is segmented on the original image (or the image acquired without the bandpass filter) by setting a brightness threshold or employing an adaptive segmentation algorithm.

[0039] S104, extract the geometric features of the keyhole region to obtain the geometric features of the keyhole corresponding to the keyhole region.

[0040] In some embodiments, the geometric features include the keyhole area, the equivalent diameter of the keyhole, the centroid position of the keyhole, and the aspect ratio of the keyhole. Extracting geometric features from the keyhole region to obtain the geometric features of the keyhole corresponding to the keyhole region includes: obtaining the total number of first pixels in the keyhole region and using the total number of first pixels as the keyhole area; calculating the equivalent diameter of the keyhole based on the keyhole area; obtaining the centroid position of the keyhole in the keyhole region; and calculating the length-to-width ratio of the minimum bounding rectangle of the keyhole region to obtain the aspect ratio of the keyhole.

[0041] Specifically, keyhole area : Total number of pixels in the keyhole area. Equivalent diameter of the keyhole. Keyhole centroid position : Used to describe the positional offset of the keyhole in the molten pool. Keyhole aspect ratio : The ratio of the length to the width of the smallest bounding rectangle of the keyhole area, reflecting the degree of elongation of its shape.

[0042] S105, perform photometric feature extraction on the plasma plume region to obtain the photometric features of the plasma plume corresponding to the plasma plume region.

[0043] In some embodiments, the photometric features include the plasma plume area, the average plasma gray value, the plasma gray standard deviation, and the number of splashes per unit time. Photometric feature extraction of the plasma plume region to obtain the photometric features of the plasma plume corresponding to the plasma plume region includes: obtaining the total number of second pixels in the plasma plume region and using the total number of second pixels as the plasma plume area; calculating the average brightness of all pixels in the plasma plume region to obtain the average plasma gray value; calculating the plasma gray standard deviation based on the average brightness; and calculating the number of splashes per unit time based on the splash detection algorithm.

[0044] Specifically, the area of ​​the plasma plume is Plasma average gray value : Represents the average brightness of all pixels within the feather region. Plasma grayscale standard deviation. This reflects the uniformity of brightness within the plume. Splatter particles are detected using splatter detection algorithms (such as background subtraction or morphological processing), and the number of splatters per unit time is counted. .

[0045] S106, dynamically extract the geometric and photometric features corresponding to the N frames of images within a window to obtain a spatiotemporal feature vector characterizing the dynamic behavior of the images within the window.

[0046] In some embodiments, in a length of Frames (e.g.) A sliding time window (corresponding to 20 milliseconds) Within this context, for each of the extracted temporal features (e.g., keyhole area sequence)... ), calculate its statistics to characterize dynamic behavior: ; ; ; in, The mean, Standard deviation, For the spectrum, For any geometric or photometric feature, The dominant frequency with the highest power spectral density and energy concentration are extracted from the spectrum. The stable oscillation frequency of the keyhole is an important basis for judging its state.

[0047] Finally, at the moment All geometric, photometric, and dynamic features calculated within the time window are concatenated to form a high-dimensional spatiotemporal feature vector: This spatiotemporal feature vector comprehensively and quantitatively describes the "spatiotemporal state" of the welding process at the current moment.

[0048] S107, input the spatiotemporal feature vector corresponding to each window into the preset temporal deep learning model to obtain the defect identification result of the welding area.

[0049] The method for obtaining defect identification results in the welding area is described in detail in subsequent embodiments, and will not be repeated here to avoid repetition.

[0050] Compared with related technologies, the embodiments of this application have at least the following advantages: By continuously acquiring images of the welding area within a preset time period, a time-series image stream is obtained. Then, the time-series image stream is shifted through a sliding window to obtain multiple windows, each containing N consecutive frames of welding area images. For each frame of the same window, geometric features of the keyhole region and photometric features of the plasma plume region are extracted. Then, dynamic features are extracted from all geometric and photometric features of the same window, thereby constructing a high-dimensional spatiotemporal feature vector that integrates geometric, photometric, and dynamic features. This spatiotemporal feature vector provides much richer process information than a single feature or indirect physical quantity (such as absorptivity). This multimodal information fusion greatly enhances the ability of the time-series deep learning model to represent complex welding processes, enabling the model to more accurately distinguish between normal states and various defect states. Especially for internal defects with complex formation mechanisms, such as porosity and lack of fusion, the recognition accuracy is significantly improved, thus enhancing the accuracy of laser welding defect identification. Furthermore, by capturing the keyhole and plasma dynamics behavior, unstable signs in the early stages of defect formation can be identified. For example, before burn-through occurs, the model can detect the continuous expansion and drastic fluctuations of the keyhole area; in areas with dense porosity formation, abnormal changes in the keyhole oscillation frequency can be observed. This enables the laser welding defect identification method of this application not only to perform post-processing identification but also to provide early warning, gaining valuable time for intervention. The entire identification process can achieve millisecond-level response on a high-performance computing platform, meeting the real-time requirements of industrial production.

[0051] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the steps of an embodiment of the laser welding defect identification method of this application. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted. This laser welding defect identification method can be applied to the aforementioned laser welding defect identification device, but is not limited thereto, and the embodiments of this application do not limit it in this regard.

[0052] This embodiment is a detailed description of the aforementioned embodiment, mainly illustrating how to obtain the defect identification result of the welding area based on the spatiotemporal feature vector corresponding to each window. In this way, multiple common welding defects (such as porosity, burn-through, collapse, lack of fusion, etc.) can be classified and identified simultaneously, providing operators with more specific and instructive diagnostic information, facilitating targeted adjustment of process parameters, and achieving precise quality control.

[0053] The specific process of this embodiment is as follows: Figure 2 As shown, it includes the following steps: S201: During the laser welding process, images of the welding area are continuously acquired within a preset time period to obtain a time-series image stream.

[0054] S202, perform a sliding window translation on the time series image stream to obtain multiple windows, where each window includes N frames of images, where N is an integer greater than 1.

[0055] S203, for each frame of an image within a window, determines the keyhole region and plasma plume region in the image.

[0056] S204, extract the geometric features of the keyhole area to obtain the geometric features of the keyhole corresponding to the keyhole area.

[0057] S205, perform photometric feature extraction on the plasma plume region to obtain the photometric features of the plasma plume corresponding to the plasma plume region.

[0058] S206, Dynamic feature extraction is performed on the geometric and photometric features corresponding to N frames of images within a window to obtain a spatiotemporal feature vector characterizing the dynamic behavior of the images within the window.

[0059] S201 to S206 of this embodiment are similar to S101 to S106 of the previous embodiment. To avoid repetition, they will not be described again here.

[0060] S207, construct a spatiotemporal feature vector sequence based on the spatiotemporal feature vector corresponding to each window.

[0061] In some embodiments, assuming that M windows are obtained after sliding window translation of the time series image stream, the spatiotemporal feature vector sequence generated in real time will be used. As input, it is fed into a pre-trained temporal deep learning model for defect classification.

[0062] S208. Input the spatiotemporal feature vector sequence into the temporal deep learning model to obtain the probability distribution vector. The probability distribution vector represents the probability of different defect categories existing in the welding area.

[0063] Specifically, a temporal deep learning model includes an input layer, a one-dimensional convolutional layer, a long short-term memory (LSM) network layer, a fully connected layer, and an output layer. The way a temporal deep learning model outputs a probability distribution vector includes: the input layer receives a sequence of spatiotemporal feature vectors; the one-dimensional convolutional layer slides along the time dimension, automatically learning the local correlation patterns between elements within the spatiotemporal feature vector sequence to obtain the target feature vector sequence; the LSM network layer receives the target feature vector sequence, captures the long-term dependencies within it, and learns the evolution of the welding process from the past to the present; the fully connected layer integrates information from the final hidden state of the LSM network layer and outputs the probability distribution vector through the output layer.

[0064] In some embodiments, the probability distribution vector is ,in, This indicates that the current welding state is judged as the first... Class defects (such as) This is normal. For the risk of porosity, The probability of a defect category (such as burn-through risk) is calculated. When the probability of a certain defect category exceeds a preset threshold, the system issues an alarm or triggers a corresponding control command.

[0065] In some embodiments, training a temporal deep learning model requires a precisely labeled dataset. Extensive welding experiments covering normal welding and welding with various defects are conducted. A coaxial vision system records the entire process video, and post-weld inspections, such as X-ray flaw detection and metallographic analysis, are used to determine the type, location, and severity of defects. Then, segments corresponding to specific defects in the video are extracted, their spatiotemporal feature vector sequences are obtained, and assigned corresponding category labels (normal, porosity, burn-through, etc.) to form training samples. The model is then trained end-to-end using the cross-entropy loss function. ;in, It is the total number of categories. It is the one-hot encoding of the real label. It is the probability predicted by the model.

[0066] To facilitate understanding, the following will be combined with... Figure 3 and Figure 4 This embodiment provides a detailed explanation of how laser welding defect identification is achieved: Please refer to this as well. Figure 3 and Figure 4 , Figure 3 This is a schematic diagram of the online identification system provided in this embodiment. Figure 4 This is a diagram of the CNN-LSTM hybrid deep learning model architecture provided in this embodiment.

[0067] The online identification system comprises a laser welding subsystem, a coaxial vision monitoring subsystem, and a data processing and decision-making subsystem. The laser welding subsystem provides the laser source, optical head, and motion platform. The coaxial vision monitoring subsystem, integrated within the laser head, includes a high-speed camera, imaging objective, bandpass filter, and a dichroic mirror that is highly transparent to the welding laser wavelength (e.g., 1070 nm) and highly reflective to the illumination source wavelength (e.g., 532 nm). The illumination source is used to illuminate the workpiece surface. The data processing and decision-making subsystem is a high-performance computer equipped with image processing algorithms and deep learning models. The high-speed camera, through reflection from the dichroic mirror, coaxially captures images of the welding area with the laser beam and transmits the video stream to the computer in real time.

[0068] This embodiment takes the online defect identification during the butt laser welding process of aluminum alloy 5052-H32 (1mm thickness) as an example, as detailed below: 1. Experimental Equipment and Parameters: A 2kW fiber laser with a wavelength of 1070nm was used. The laser head was tilted at 10°. The coaxial vision monitoring system used a Photron SA3 high-speed camera, with a frame rate of 2000 fps and a resolution of 384. 320 pixels. The illumination source is a 50W 532nm LED light source, coupled with a bandpass filter of the corresponding wavelength. The data processing computer is equipped with an NVIDIA TITAN RTX GPU. Normal welds, porosity defect welds, and burn-through defect welds are induced by adjusting the laser power (1500W-1800W) and welding speed (7.0 mm / s - 9.0 mm / s).

[0069] 2. Data Acquisition and Labeling: A total of 30 welding experiments were conducted, recording approximately 10 minutes of high-speed video. After welding, all welds underwent X-ray non-destructive testing to accurately locate porosity and burn-through defects. The X-ray results were synchronized with the video timestamps, and the video data was labeled frame by frame. Labels were divided into three categories: 0 - Normal, 1 - Porosity, and 2 - Burn-through.

[0070] 3. Feature Extraction and Model Training: Setting the Length of the Sliding Time Window Frame (10 ms), length of feature vector sequence (That is, the model reviews historical information from the past 100 ms). Extracted spatiotemporal feature vectors. It includes 24 dimensions, specifically: keyhole area, equivalent diameter, centroid X / Y coordinates, aspect ratio, and plasma plume area, average gray level, and gray level standard deviation (a total of 8 basic features). The mean, standard deviation, and dominant frequency of these 8 features are calculated within a time window. (Dynamic features). The constructed CNN-LSTM model consists of two one-dimensional convolutional layers, one stacked LSTM network (128 hidden units per layer), and a 3-class softmax output layer. The labeled dataset is divided into training, validation, and test sets in a 7:2:1 ratio. The Adam optimizer is used with an initial learning rate of 0.001, and training is performed for 200 epochs.

[0071] 4. Results and Analysis: The trained model was deployed on the test platform for online recognition. In one test case, when the welding process transitioned from a steady state to a state with frequent porosity, the system monitored the standard deviation of the keyhole area. From 0.05 mm Jumped to 0.2 mm And its oscillation frequency It drifts from a stable frequency around 200 Hz to a wide frequency band of 500-800 Hz. Simultaneously, the average gray level of the plasma... The fluctuations also intensified significantly. These significant changes in the spatiotemporal feature vectors were captured by the CNN-LSTM model, and the probability of the "porosity" category output by the model rapidly increased from less than 10% to over 95%, successfully achieving online identification of porosity defects. In another case of burn-through defects, approximately 50 milliseconds before the weld actually burned through, the model detected the mean area of ​​the keyhole. A sustained, monotonous increase occurred, far exceeding the normal range, while the plasma area... The probability of the "burn-through" category is increased dramatically. Based on this, the model increases the probability of the "burn-through" category to 98%. On the entire test set, the overall recognition accuracy of the method of this invention reaches 96.3%, which is a significant improvement in performance compared to CNN models that only use single-frame geometric features (accuracy 85.1%) and indirect methods that predict absorptivity.

[0072] Compared with related technologies, the embodiments of this application have at least the following advantages: By continuously acquiring images of the welding area within a preset time period, a time-series image stream is obtained. Then, the time-series image stream is shifted through a sliding window to obtain multiple windows, each containing N consecutive frames of welding area images. For each frame of the same window, geometric features of the keyhole region and photometric features of the plasma plume region are extracted. Then, dynamic features are extracted from all geometric and photometric features of the same window, thereby constructing a high-dimensional spatiotemporal feature vector that integrates geometric, photometric, and dynamic features. This spatiotemporal feature vector provides much richer process information than a single feature or indirect physical quantity (such as absorptivity). This multimodal information fusion greatly enhances the ability of the time-series deep learning model to represent complex welding processes, enabling the model to more accurately distinguish between normal states and various defect states. Especially for internal defects with complex formation mechanisms, such as porosity and lack of fusion, the recognition accuracy is significantly improved, thus enhancing the accuracy of laser welding defect identification. Furthermore, by capturing the keyhole and plasma dynamics behavior, unstable signs in the early stages of defect formation can be identified. For example, before burn-through occurs, the model can detect the continuous expansion and drastic fluctuations of the keyhole area; in areas with dense porosity formation, abnormal changes in the keyhole oscillation frequency can be observed. This enables the laser welding defect identification method of this application not only to perform post-processing identification but also to provide early warning, gaining valuable time for intervention. The entire identification process can achieve millisecond-level response on a high-performance computing platform, meeting the real-time requirements of industrial production.

[0073] Based on the same idea as the laser welding defect identification method in the above embodiments, this application also provides a laser welding defect identification device, which can be used to perform the above-described laser welding defect identification method. For ease of explanation, the structural schematic diagram of the laser welding defect identification device embodiment only shows the parts related to the embodiments of this application. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0074] like Figure 5 As shown, the laser welding defect identification device 50 includes an acquisition module 501, a sliding window module 502, a determination module 503, a geometric feature extraction module 504, a photometric feature extraction module 505, a dynamic feature extraction module 506, and an input module 507. In some embodiments, the above modules can be programmable software instructions stored in a memory and executable by a processor. It is understood that in other embodiments, the above modules can also be program instructions or firmware embedded in a processor.

[0075] The acquisition module 501 is used to continuously acquire images of the welding area within a preset time period during the laser welding process to obtain a time-series image stream; The sliding window module 502 is used to perform sliding window translation on the time series image stream to obtain multiple windows, where each window includes N frames of images, where N is an integer greater than 1; The determination module 503 is used to determine the keyhole region and plasma plume region in each frame of an image within a window. The geometric feature extraction module 504 is used to extract geometric features from the keyhole region to obtain the geometric features of the keyhole corresponding to the keyhole region. The photometric feature extraction module 505 is used to extract photometric features from the plasma plume region to obtain the photometric features of the plasma plume corresponding to the plasma plume region. The dynamic feature extraction module 506 is used to extract dynamic features from the geometric and photometric features corresponding to N frames of images within a window, respectively, to obtain a spatiotemporal feature vector characterizing the dynamic behavior of the images within the window. The input module 507 is used to input the spatiotemporal feature vector corresponding to each window into a preset temporal deep learning model to obtain the defect identification result of the welding area.

[0076] The laser welding defect identification device 50 provided in the above embodiments can realize the technical solutions described in the above laser welding defect identification method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above laser welding defect identification method embodiments, and will not be repeated here.

[0077] Please refer to Figure 6 , Figure 6 This is a schematic diagram of an embodiment of the electronic device of this application. In this embodiment of the invention, the electronic device 600 includes a processor 601, a memory 602, and a display 603. Figure 6 Only some components of the electronic device 600 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0078] In some embodiments, processor 601 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 602 or process data, such as the laser welding defect identification method of the present invention.

[0079] In some embodiments, processor 601 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 601 may be local or remote. In some embodiments, processor 601 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, intranet, multi-cloud, etc., or any combination thereof.

[0080] In some embodiments, memory 602 may be an internal storage unit of electronic device 600, such as a hard disk or memory of electronic device 600. In other embodiments, memory 602 may also be an external storage device of electronic device 600, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 600.

[0081] Furthermore, the memory 602 may include both internal storage units of the electronic device 600 and external storage devices. The memory 602 is used to store application software and various types of data installed on the electronic device 600.

[0082] In some embodiments, display 603 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 603 is used to display information from electronic device 600 and to display visual user applications. Components 601-603 of electronic device 600 communicate with each other via a system bus.

[0083] In one embodiment, when the processor 601 executes the laser welding defect identification program in the memory 602, the following steps can be implemented: During laser welding, images of the welding area are continuously acquired within a preset time period to obtain a time-series image stream. The time-series image stream is shifted using a sliding window to obtain multiple windows, each of which includes N frames of the images, where N is an integer greater than 1. For each frame of the image within a window, determine the keyhole region and plasma plume region in the image; Geometric features are extracted from the keyhole region to obtain the geometric features of the keyhole corresponding to the keyhole region; Photometric features are extracted from the plasma plume region to obtain the photometric features of the plasma plume corresponding to the plasma plume region. Dynamic feature extraction is performed on the geometric features and photometric features corresponding to the N frames of the image within a window to obtain a spatiotemporal feature vector characterizing the dynamic behavior of the image within the window; The spatiotemporal feature vector corresponding to each window is input into a preset temporal deep learning model to obtain the defect identification result of the welding area.

[0084] It should be understood that when the processor 601 executes the laser welding defect identification program in the memory 602, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0085] Furthermore, this embodiment of the invention does not specifically limit the type of electronic device 600 mentioned. Electronic device 600 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the invention, electronic device 600 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0086] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the laser welding defect identification method provided in the above-described method embodiments.

[0087] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0088] The laser welding defect identification method, apparatus, electronic device, and computer-readable storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for identifying defects in laser welding, characterized in that, include: During laser welding, images of the welding area are continuously acquired within a preset time period to obtain a time-series image stream. The time-series image stream is shifted using a sliding window to obtain multiple windows, each of which includes N frames of the images, where N is an integer greater than 1. For each frame of the image within a window, determine the keyhole region and plasma plume region in the image; Geometric features are extracted from the keyhole region to obtain the geometric features of the keyhole corresponding to the keyhole region; the geometric features include the keyhole area, the equivalent diameter of the keyhole, the position of the centroid of the keyhole, and the aspect ratio of the keyhole. Photometric features are extracted from the plasma plume region to obtain the photometric features of the plasma plume corresponding to the plasma plume region; the photometric features include plasma plume area, average plasma gray value, plasma gray standard deviation, and number of sputterings per unit time. Dynamic feature extraction is performed on the geometric features and photometric features corresponding to N frames of images within a window to obtain a spatiotemporal feature vector characterizing the dynamic behavior of the images within the window. This includes calculating the mean, standard deviation, and spectrum of the geometric features or photometric features corresponding to N frames of images within a window according to the following formula: ; ; ;in, The mean is... The standard deviation is... For the spectrum, For any of the geometric features or photometric features; extract the dominant frequency with the highest power spectral density from the spectrum; construct the spatiotemporal feature vector based on the mean, standard deviation, and dominant frequency corresponding to each geometric feature or photometric feature; The process involves inputting the spatiotemporal feature vector corresponding to each window into a preset temporal deep learning model to obtain the defect identification result of the welding area, including: constructing a spatiotemporal feature vector sequence based on the spatiotemporal feature vector corresponding to each window; inputting the spatiotemporal feature vector sequence into the temporal deep learning model to obtain a probability distribution vector, wherein the probability distribution vector represents the probability of different defect categories existing in the welding area; The temporal deep learning model includes an input layer, a one-dimensional convolutional layer, a long short-term memory network layer, a fully connected layer, and an output layer. The temporal deep learning model outputs the probability distribution vector in the following ways: The input layer receives the spatiotemporal feature vector sequence; The one-dimensional convolutional layer slides along the time dimension, automatically learning the local correlation patterns between elements within the spatiotemporal feature vector sequence to obtain the target feature vector sequence. The long short-term memory network layer receives the target feature vector sequence, captures the long-term dependencies in the target feature vector sequence, and learns the evolution of the welding process from the past to the present. The fully connected layer integrates information about the final hidden state of the long short-term memory network layer and outputs the probability distribution vector through the output layer.

2. The laser welding defect identification method according to claim 1, characterized in that, The step of extracting geometric features from the keyhole region to obtain the geometric features of the keyhole corresponding to the keyhole region includes: Obtain the total number of first pixels in the keyhole area, and use the total number of first pixels as the keyhole area; Calculate the equivalent diameter of the keyhole based on the keyhole area; Obtain the centroid position of the keyhole in the keyhole region; Calculate the ratio of the length to the width of the smallest bounding rectangle of the keyhole region to obtain the aspect ratio of the keyhole.

3. The laser welding defect identification method according to claim 1, characterized in that, The step of extracting photometric features from the plasma plume region to obtain the photometric features of the plasma plume corresponding to the plasma plume region includes: Obtain the total number of second pixels in the plasma plume region, and use the total number of second pixels as the plasma plume area; Calculate the average brightness of all pixels within the plasma feather region to obtain the average grayscale value of the plasma; Calculate the plasma grayscale standard deviation based on the average brightness; The number of splashes per unit time is calculated based on the splash detection algorithm.

4. The laser welding defect identification method according to claim 1, characterized in that, The continuous acquisition of images of the welding area within a preset time period during the laser welding process includes: During laser welding, images of the welding area are continuously acquired at a preset frame rate and a preset resolution within a preset time period.

5. A laser welding defect identification device, characterized in that, include: The module includes an acquisition module, a sliding window module, a determination module, a geometric feature extraction module, a photometric feature extraction module, a dynamic feature extraction module, and an input module. The acquisition module is used to continuously acquire images of the welding area within a preset time period during the laser welding process to obtain a time-series image stream; The sliding window module is used to perform sliding window translation on the time series image stream to obtain multiple windows, wherein each window includes N frames of the image, where N is an integer greater than 1; The determining module is used to determine the keyhole region and plasma plume region in each frame of the image within a window; The geometric feature extraction module is used to extract geometric features from the keyhole region to obtain the geometric features of the keyhole corresponding to the keyhole region; the geometric features include the keyhole area, the equivalent diameter of the keyhole, the position of the centroid of the keyhole, and the aspect ratio of the keyhole. The photometric feature extraction module is used to extract photometric features from the plasma plume region to obtain the photometric features of the plasma plume corresponding to the plasma plume region; the photometric features include plasma plume area, average plasma gray value, plasma gray standard deviation, and number of splashes per unit time. The dynamic feature extraction module is used to perform dynamic feature extraction on the geometric features and photometric features corresponding to N frames of the image within a window, respectively, to obtain a spatiotemporal feature vector characterizing the dynamic behavior of the image within the window, including: calculating the mean, standard deviation, and spectrum of the geometric features or photometric features corresponding to N frames of the image within a window according to the following formula: ; ; ;in, The mean is... The standard deviation is... For the spectrum, For any of the geometric features or photometric features; extract the dominant frequency with the highest power spectral density from the spectrum; construct the spatiotemporal feature vector based on the mean, standard deviation, and dominant frequency corresponding to each geometric feature or photometric feature; The input module is used to input the spatiotemporal feature vector corresponding to each window into a preset temporal deep learning model to obtain the defect identification result of the welding area, including: constructing a spatiotemporal feature vector sequence based on the spatiotemporal feature vector corresponding to each window; inputting the spatiotemporal feature vector sequence into the temporal deep learning model to obtain a probability distribution vector, wherein the probability distribution vector represents the probability of different defect categories existing in the welding area; The temporal deep learning model includes an input layer, a one-dimensional convolutional layer, a long short-term memory network layer, a fully connected layer, and an output layer. The temporal deep learning model outputs the probability distribution vector in the following ways: The input layer receives the spatiotemporal feature vector sequence; The one-dimensional convolutional layer slides along the time dimension, automatically learning the local correlation patterns between elements within the spatiotemporal feature vector sequence to obtain the target feature vector sequence. The long short-term memory network layer receives the target feature vector sequence, captures the long-term dependencies in the target feature vector sequence, and learns the evolution of the welding process from the past to the present. The fully connected layer integrates information about the final hidden state of the long short-term memory network layer and outputs the probability distribution vector through the output layer.

6. An electronic device, the electronic device comprising a processor and a memory, characterized in that, The memory is used to store instructions, and the processor is used to call the instructions in the memory to cause the electronic device to execute the laser welding defect identification method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the laser welding defect identification method as described in any one of claims 1 to 4.

Images