Method for in-situ inspection of welding quality

Abstract

A method performs in-situ inspection of weld quality. The method acquires multiple sequential images of an ongoing welding process and generates a multidimensional data input based on the multiple sequential images and / or one or more welding process control parameters. The welding process control parameters may include (i) shielding gas flow rate, temperature, and pressure; (ii) voltage, current, wire feed speed, and temperature (if applicable); (iii) part preheat / interpass temperature; and (iv) relative speed between the part and the welding torch. The method generates defect probability and analytical information by applying one or more computer vision techniques to the multidimensional data input. The analytical information includes predictive insights into quality characteristics of the ongoing welding process. The method generates a 3D visualization of one or more as-welded areas based on the analytical information and the multiple sequential images. The 3D visualization displays quality features for virtual inspection and / or for determining weld quality.

Description

Technical Field

[0001] The disclosed embodiments generally relate to welding, and more particularly to systems, methods, and user interfaces for acquisition by in-situ sensors and inspection based on digital machine learning / deep learning models.

Background Art

[0002] Welding technology has a significant impact on the manufacturing industry and the economy as a whole. Advancements in welding technology (e.g., robotic welding) provide cost efficiency and consistency. Welding quality is important for the safety and integrity of systems. The manufacture of components of safety-critical systems such as nuclear reactor pressure vessels is typically required by strict requirements and design criteria. Conventionally, such requirements are verified by expensive non-destructive examination (NDE) after the welding operation is completed, or by prequalification of the welding process (to predict welding quality). After the welding process is completed, periodic repairs (e.g., replacement or welding of defective parts) are performed to ensure quality, but the cause of the defect may not be known. Conventional techniques for welding quality control are error-prone and expensive.

Summary of the Invention

[0003] In addition to the problems described in the background art, there are other reasons why there is a need to improve the systems and methods for inspecting welding quality. For example, existing techniques rely on post-mortem analysis of welding defects, so there is no context information for proper root cause analysis. Some techniques can only be applied to a limited range of welding processes. Conventional systems for welding inspection rely on process method approval, NDE post-weld inspection, or regression techniques using welding process parameters such as voltage, torch speed, current, gas flow rate, etc., but such conventional methods do not regress well to desirable quality characteristics. This disclosure describes systems and methods that address at least some of the drawbacks of conventional methods and systems.

[0004] Based on some embodiments, the present disclosure uses computer vision, machine learning, and / or statistical modeling to build a digital model for in-situ inspection of welding quality (also called in-situ inspection, i.e., inspection of welding quality while welding is in progress).

[0005] Visualization is basically based on in-situ image signals or other processed signals and is obtained as a result of computer vision with predictive insights from machine learning / deep learning algorithms.

[0006] Based on some embodiments, the present invention uses one or more cameras as sensors to acquire continuous images (e.g., multiple still images or videos) over a weld of a welding event (e.g., events such as melting, cooling, and seam formation of the base material and filler). The continuous images are processed as a 3D data array by computer vision and machine learning / deep learning techniques to generate appropriate analysis for determining welding quality, a 3D visual representation of the as-welded region where quality characteristics appear for virtual inspection, and / or predictive insights into the location and extent of quality features. In some embodiments, images of the ongoing welding process are processed using trained computer vision and machine learning / deep learning algorithms to generate dimensionally accurate visualization and defect characterization. In some embodiments, the computer vision and machine learning / deep learning algorithms are trained to determine welding quality based on images of a good puddle shape.

[0007] According to some embodiments, the method is executed on a computing system. Generally, the computing system includes a single computer or workstation, or multiple computers, each computer or workstation having one or more CPU and / or GPU processors and memory. The machine learning modeling method of the embodiments generally does not require a computing cluster or supercomputer.

[0008] In some embodiments, the computing system includes one or more computers. Each computer includes one or more processors and memory. The memory stores one or more programs configured to be executed by one or more processors. The one or more programs include instructions for executing any of the methods disclosed herein.

[0009] In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by a computing system, the computing system having one or more computers, each computer having one or more processors and memory. The one or more programs include instructions for executing any of the methods disclosed herein.

[0010] Thus, the disclosed method and system facilitate in-situ inspection of the welding process. The considerations, examples, principles, compositions, structures, features, arrangements, and processes described in this disclosure are applicable, adaptable, and implementable to the welding process.

[0011] To better understand the disclosed system and method, and additional systems and methods, reference should be made to the following examples in conjunction with the following drawings, wherein like reference numerals refer to corresponding parts throughout the drawings.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 3C

Figure 4A

Figure 4B

Figure 4C

Figure 5A

Figure 5B

Figure 6A

Figure 6B

Figure 7A

Figure 7B

Figure 7C

Figure 7D

Figure 7E

Figure 7F

Figure 7G

Figure 7H

Figure 7I

Figure 8A

Figure 8B

Figure 9

Figure 10A

Figure 10B

DETAILED DESCRIPTION OF THE INVENTION

[0013] Here, reference is made to embodiments, examples of which are illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that these specific details are not required to practice the present invention.

[0014] Figure 1 is a block diagram of a system 100 for in-situ inspection of a welding process using a digital data model, based on several embodiments. The welding device 102 is monitored by one or more camera devices 104, each device 104 including one or more image sensors 106 and one or more image processing devices 108. Data collected by the camera devices is communicated to an in-situ inspection server 112 using a communication network 110. The welding device 102 uses a set 118 of welding parameters, which can be dynamically updated by the in-situ inspection server 112.

[0015] The in-situ inspection server 112 uses several standard computer vision processing algorithms 114 and several machine learning / deep learning data models 115.

[0016] In this process, images during welding operations are captured in-situ and standard image processing techniques are applied to enhance features (e.g., Gaussian blur, edge detection between the electrode and the welding pool, signal to noise filtering, angle correction, etc.). In this process, temporal cross-correlation is used to geometrically align image stacks or video frames. In some embodiments, this information is provided to one or more mounted robotic cameras for accurate image acquisition. The system converts the temporal trend of the image into a stationary signal by differentiating the image over time. This system uses 3D convolution (e.g., pixel position, intensity, color / spectral band, etc.) to train a convolutional neural network with consecutive and delayed image batches. Based on this, the machine learning / deep learning data model 115 outputs the probability of a specific event (presence or absence of defects or type of defects).

[0017] The parameter data model 116 identifies abnormal parts of the signal. In the signal noise processing of conventional monitored welding parameters (e.g., voltage along the time series), it is not possible to indicate defects in welding quality. This process is carried out in the following series of steps. (i) Convert the analog signal to digital. (ii) Train a temporal convolutional neural network with a sliding window and a gated activation function to learn typical signal patterns over a large number (e.g., millions) of time series data points. (iii) Minimize the cross-entropy loss function. (iv) Obtain the difference between the parameter data stream and the learned data stream. (v) Use kernel density estimation to find abnormal parts of the signal.

[0018] The parameter data model control unit 116 feeds back to the operation and / or controls the welding parameters in order to maintain quality. The convolutional network weights the parameters so as to minimize the loss function. The weights include information from an image regarding a main characteristic indicating a defect. The operation is advanced to show the main defect characteristics by providing a visualization of the normalized gradient of the weights. These weights are temporally displayed along a temporal image batch so as to temporally identify the location of the defect. Different portions such as the intensity, shape, and hue of the image are shown by these weights. The parameter data model control 116 collects a data set of all defect indications. This is input into a statistical model (e.g., Poisson regression) to map the valid and invalid welding parameter space.

[0019] In some embodiments, the parameter data model control unit 116 warns of defects that can occur in the topology (form, topology). A high-fidelity topology can be provided for automated welding to avoid defects.

[0020] FIG. 2 is a block diagram showing a computing device 200 according to some embodiments. Various examples of the computing device 200 include high-performance clusters (HPC) of servers, supercomputers, desktop computers, cloud servers, and other computing devices. The computing device 200 typically includes one or more processing units / cores (CPUs and / or GPUs) 202 for performing processing operations by executing modules, programs, and / or instructions stored in the memory 214, one or more network or other communication interfaces 204, the memory 214, and one or more communication buses 212 for interconnecting these components. The communication bus 212 may include circuitry for interconnecting and controlling communication between the components of the system.

[0021] Computing device 200 may include a user interface 206, and the user interface 206 may include a display device 208 and one or more input devices or mechanisms 210. In some embodiments, the input device / mechanism includes a keyboard. In some embodiments, the input device / mechanism includes a "soft" keyboard that is selectively displayed on the display device 208 and enables a user to "press keys" displayed on the display 208. In some embodiments, the display 208 and the input device / mechanism 210 include a touch screen display (also referred to as a touch sensitive display).

[0022] In some embodiments, memory 214 includes high speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices. In some embodiments, memory 214 includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. In some embodiments, memory 214 includes one or more memory devices located remote from the GPU / CPU 202. Memory 214, or alternatively the non-volatile memory device within memory 214, includes a non-transitory computer readable storage medium. In some embodiments, memory 214, or the computer readable storage medium of memory 214, stores the following programs, modules, and data structures, or subsets thereof. That is, an operating system 216 that includes procedures for handling various basic system services and performing hardware dependent tasks, and one or more communication network interfaces 204 (wired or wireless) and a communication module 218 used to connect the computing device 200 to other computers and devices via one or more communication networks such as the Internet, other wide area networks, local area networks, metropolitan area networks. For in-situ inspection, a data visualization application or module 220 for displaying the visualization results of welding defects, an input / output user interface processing module (not shown) that allows a user to specify parameters and control variables, the in-situ inspection engine 112 described above with respect to FIG. 1, a feature vector 246 used by the machine learning / deep learning model 115, a machine learning / deep learning / regression model 115, stores, or stores a subset of these.

[0023] Each of the executable modules, applications, or sets of procedures described above can be stored in one or more of the memory devices described above and corresponds to a set of instructions for performing the functions described above. The identified modules or programs described above (i.e., sets of instructions) need not be executed as separate software programs, procedures, or modules, and in various embodiments, various subsets of these modules can be combined or rearranged. In some embodiments, the memory 214 stores a subset of the modules and data structures described above. Further, the memory 214 may store additional modules or data structures not described above.

[0024] FIG. 2 shows a computing device 200, but FIG. 2 is not intended as a structural diagram of some of the embodiments described in the present disclosure, but rather as a functional description of various features that may exist. In fact, as will be appreciated by those skilled in the art, the separately shown items can be combined and some items can be separated.

[0025] In some embodiments, although not shown, the memory 214 also includes modules for training and executing the models described above with reference to FIG. 1. Specifically, in some embodiments, the memory 214 also includes a stochastic sampling module, a machine learning model 115, a coding framework, one or more convolutional neural networks, a statistical support package, and other images, signals, or related data. Illustrations of welding process and evaluation of welding quality

[0026] According to some embodiments, the techniques disclosed herein are applicable to a wide range of welding processes. For example, the techniques can be used to inspect the quality of gas tungsten arc welding or GTAW (also known as tungsten inert gas welding or TIG), plasma arc welding, laser welding, electron beam welding, shielded metal, and gas metal welding, automated and / or manual welding, pulsed welding, and submerged welding. In some embodiments, the techniques are applicable to operations across multiple facilities and / or two or more types of welding (e.g., GTAW where the welding torch moves across a fixed portion such as in many build-up welds or some straight-line welds, and GTAW where the welding torch is fixed and the part rotates such as in circle seam welds or some build-up welds). In some embodiments, the techniques are used to simultaneously inspect the quality of welds for a large number of welds (e.g., where a particular steam generator has 257 thick welds with strict inspection criteria and a high rejection rate).

[0027] FIG. 3A shows an exemplary platform and crane apparatus 300 for welding large structures, according to some embodiments.

[0028] Figure 3B is an illustration of a welding process 302 based on some embodiments. This illustration shows a robotic arm welder 304 and an illustration 306 of the welding process. Conventionally, in robotic welding, a welding technician monitors the welding fusion and the deposition of filler material (e.g., via a video monitor). The technician discovers abnormalities and determines the welding quality based on experience and observation. Conventional systems were unable to acquire welding process data or did not use the acquired data for quality inspection. In certain situations, pre-inspection quality control is performed using a process approved based on a pre-production mock-up, in which the process parameters are determined. Many conventional systems require human monitoring, are subjective and result-variable, and / or can detect only a small number of defects. Figure 3C is another exemplary welding process 308 based on some embodiments. This illustration shows a dual welding station 310 with a shell-penetrating nozzle and a single platform 312 having a desk and storage space and connecting to the two stations.

[0029] In conventional systems, mock-ups are used for setting process control parameters and it is done by trial and error. The welding data sheet specifies the parameters to be tried first. The parameters are repeatedly optimized based on experimental results. Some embodiments use welding inspection techniques such as radiographic inspection (sensitive to, e.g., corrosion, thickness changes, voids, cracks, material density changes), ultrasonic inspection (a method for detecting defects on the surface or back surface of materials and measuring the wall thickness of round materials such as tubes and pipes), magnetic particle inspection (used for detecting defects on the surface / nearby of ferromagnetic materials), and visual inspection (a visual confirmation of integrity, cracks, and uniformity). In some embodiments, a dye penetrant test (dye PT) is performed to inspect for surface flaws on-site. The PT may be performed after several layers are completed, after which welding continues.

[0030] Some embodiments use machine vision to inspect weld quality during an ongoing welding process. Some embodiments use deep learning techniques, where the input parameters need not be explicitly defined and the algorithm automatically derives the parameters. Some embodiments use machine vision and / or image processing techniques to obtain a non-linear correlation of weld quality (e.g., as applied to a physically linear weld path). Some embodiments use the techniques described herein for additive manufacturing (also referred to as 3D printing) to acquire images (e.g., layer-by-layer) using built-in sensors. Some embodiments perform real-time monitoring and defect identification when or immediately after a defect occurs. In some embodiments, limited image parameters (e.g., the shape of the weld pool and / or the box boundary around the shape) are used. Some embodiments process images based on trained computer vision and machine learning / deep learning algorithms to generate intelligent image reconstruction and quality prediction and / or to generate a dimensionally accurate visual and quantitative characterization of welding defects during (or upon completion of) the welding process. Illustrations of shape analysis, laser scanning, and neural network

[0031] Some embodiments use one or more optical cameras with mechanical shutters and a laser to acquire an image of the surface of the weld pool. Some embodiments apply image processing and machine learning algorithms to the complete shape and / or image of the weld pool rather than approximating the size of the weld pool. For example, conventional systems approximate the size of the weld pool section with the dimensions of a two-dimensional bounding box (width, height) and / or define the shape of the melt pool with an angle included in the tail. In conventional systems, machine learning algorithms are trained using limited scalar attributes extracted from the image. FIGS. 4A-4C show illustrations of the shape of a melt pool processed or analyzed using the techniques described in this disclosure, based on some embodiments.

[0032] Some embodiments use laser scanning. A point laser irradiates a laser beam on a welding surface, and an image sensor captures light through one or more lens filters to obtain an image of the welding surface. Post-processing is performed on the image to model the surface. Some embodiments use two-dimensional cross-sectional laser scanning to model the welding surface. Some embodiments identify variations in surface shape with laser scanning and detect or estimate defects beneath the surface. Some embodiments utilize the deviation of the measured value of the welding deposition amount to identify voids and other defects beneath the surface. Some embodiments use one or more laser profiles to enhance or determine the profile of the surface shape. Some embodiments use laser scanning in addition to, or to reinforce, other techniques described herein.

[0033] Some embodiments use a neural network to process images during welding to determine welding defects. For the purpose of in-situ welding inspection, some embodiments apply, modify, and / or discover (or search) appropriate machine learning and deep learning. Some embodiments tune or adjust hyperparameters according to the type of welding, setup, and sensor configuration.

[0034] Some embodiments use a convolutional neural network (CNN) filter to recognize the geometric features of the welding and the trained patterns of the object. Some embodiments train a CNN to recognize welding quality features of interest. Some embodiments use image processing, CNN construction, and hyperparameters with respect to voids, misalignment, undercut, porosity, variation of the welding path, and / or crack formation.

[0035] Some embodiments generate the effect of a shadow using an appropriate camera based on the imaging wavelength, acoustic device, near-infrared camera, optical camera, and laser illumination technology.

[0036] Some embodiments provide similar advantages as the techniques used in additive manufacturing (the layer-by-layer manufacturing method helps image the built state in slices). Some embodiments use a high-definition infrared (HSIR) camera and also use high frame rate imaging of the ongoing welding process to provide inspection and prediction effects similar to additive manufacturing for conventional welding processes.

[0037] Some embodiments use one or more cameras as sensors to extract continuous images (still images or videos, e.g., images of the melting, cooling, and seam formation events of the base metal and filler material) during welding. In some embodiments, the images are processed as a 3D data array by computer vision and machine learning / deep learning techniques to generate appropriate analysis for determining weld quality, a 3D visual display of the as-welded area where quality characteristics appear for virtual inspection, and / or predictive insights into the location and extent of quality features. Illustrations of defects

[0038] Figures 5A and 5B show illustrations of welding defects according to some embodiments. Figure 5A shows different types of welding defects 500 including undercut 502, normal weld 504, groove 506, sharp corner 508, porosity or inclusion 510, and misalignment 512, according to some embodiments. Figure 5B shows lack of fusion, or partial fusion weld 514, according to some embodiments. Typically, such defects require rework or scrapping, and such defects include lack of fusion, lack of penetration, porosity, cracks, and defects due to undercut. As a result, manufacturing delays occur due to replacement parts or inspection time. Illustrations of methods for in-situ inspection of welding quality

[0039] Some embodiments use one or more cameras to collect infrared, near-infrared, and / or optical images (e.g., individual images and / or videos) of the welding event arc, the electrode, and / or the weld pool to detect, infer, predict, and / or visualize welding quality characteristics of interest.

[0040] Some embodiments use computer vision (e.g., Python® OpenCV code) in combination with multiple sensor images and / or laser line profiling to detect quality defects in welding. Some embodiments clean, align, register the image data, emphasize thresholds for noise and objects, and statistically filter to represent and identify patterns and features useful for quality determination. Some embodiments use a 2D or 3D model of the weld seam or product to visualize the observed defects. Some embodiments visualize in 3D the shape and vibration changes of the molten pool and / or display the location and representation of contaminants in the weld pool. Some embodiments visualize (or display) the weld, shape, texture, size, arrangement, and contaminants immediately after welding and during cooling. Some embodiments detect and / or display arc changes in shape and intensity. Some embodiments detect and / or display electrode spatter and / or degradation, 3D profile information and patterns formed by fillet welding, and / or the integrity, voids, and separation regions of seam welding. Illustrations of machine learning and statistical modeling techniques

[0041] Some embodiments use machine learning or deep learning (e.g., TensorFlow® / Keras) to learn and interpret the acquired welding image data. In some embodiments, the algorithm converts a continuous image of the welding into a data array. Some embodiments integrate welding parameter data. Some embodiments use a convolutional neural network algorithm (e.g., TensorFlow, Keras, or a similar open-source machine learning platform) to process the welding image data. Some embodiments utilize unsupervised anomaly detection to train a model of good welds. Some embodiments flag welding signals that exceed an error threshold as anomalies to be noted. For example, the machine learning algorithm predicts that a welding error exceeds a threshold of error probability (e.g., 10%), and the corresponding welding signal is flagged as an anomaly. Some embodiments use supervised defect detection, in which the model is trained using images in a database of known defects (e.g., images of different types of induced defects generated to train or test the model).

[0042] Some embodiments use actual images of features of interest to obtain the actual boundaries, speeds, spatter, rates of change, and other parameters of the molten pool shape, train a machine learning algorithm to predict welding quality, defects, and / or provide a characterization of welding quality. FIG. 6A shows an illustration of a process for predicting welding quality using a neural network, according to some embodiments. Some embodiments input a welding image 602 into a trained neural network 604 (a neural network trained to determine welding quality using welding images such as a TIG list from NOG-B) to generate a probability estimate 606 of different types of welding defects (or no defects).

[0043] In some embodiments, the 2D or 3D digital data model of the weld digital twin is annotated for quality inspection prior to visual or instrument inspection outside the weld site. Some embodiments facilitate inspection by flagging regions of interest. Some embodiments facilitate interpretation by an inspector of anomalies or indications. Some embodiments include a system that warns an operator of events predicted or occurred during welding, such that defects can be corrected during operation (or immediately after a defect appears), reducing the overall amount of repair. Some embodiments provide statistical summaries and analysis, quantify observed features, and / or predict the characteristics of quality features that appear over time and features not directly observable by sensors without relying on post-weld inspection techniques. In some embodiments, the final weld digital twin is annotated with quality evaluations of weld defects, such as size, shape, extent, depth, type, etc., for virtual inspection.

[0044] For machine learning, some embodiments use a sequencing model to extract unmonitored arc and electrode anomalies. Some embodiments use autoregressive unsupervised anomalies from the welding image signal pattern. Some embodiments separate signals that are not part of the random noise signal and indicate events. Some embodiments facilitate manual inspection of the image and the physical weld and annotate the digital model with defect information. Some embodiments generate descriptors of the shape of the molten pool from the contour. Some embodiments utilize an unsupervised classification model (e.g., a recurrent neural network) to identify various types of welding defects. Some embodiments quantify the classification of the welding pool in a certain spatial region (e.g., build length of 1 centimeter). Some embodiments create a statistical fit based on the location of an ex-situ inspection (also called "off-site inspection") of the welding. Some embodiments train a supervised neural network to classify the type of annotated defect from video input and automatically extract engineering features.

[0045] In certain cases, some embodiments use the techniques of a probabilistic volatility model to model defects. Some embodiments combine models to remove any random components and generate abnormal components. Some embodiments use a stationary space model with an autoregressive model.

[0046] Some embodiments use WaveNet (developed by DeepMind), which is a generative recurrent neural network for audio sequences, has gates within the network, and models a layered effect. Some embodiments use a neural network for autoregressive training, which is applied instead of a video image stream of specific electrode events. Some embodiments use WaveNet in combination with an error term defined by Equation (1) below.

[0047]

Number

[0048] Some embodiments use a batched stochastic gradient descent (batched SGD) algorithm.

[0049] In some embodiments, as defined by Equation (2) below, the error term is modeled by a kernel density estimator.

[0050]

Number

[0051] Some embodiments divide out the error that fits a random pattern from this error model, as defined by Equation (3) below, and the remaining amount or what's left is the detected anomaly.

[0052]

Number

[0053] Illustrations of systems and methods for image acquisition Figure 6B shows an illustration of a camera system (or image acquisition system) 608 according to some embodiments. In some embodiments, the camera acquisition system is disposed near (e.g., G-5’), on a mounting platform fixed to a tripod or robotic arm, or fixed from above on a rigid surface, or otherwise disposed where the weld is visible. The camera system 608 collects images and / or videos of ongoing welding. In some embodiments, the weld images record patterns and behaviors of welding events such as the shape, size, intensity pattern, contour, depth, temperature gradient, changes over time, uniformity, spatter, alignment, and other hidden variables and interactions. The other hidden variables and interactions are not explicitly defined as inputs and are ultimately used to determine the relationship with the quality of the welded product. Some embodiments use a high-speed optical camera or video camera 610 (e.g., a camera capable of shooting at 200 FPS (frames per second)) and transfer the images to an image acquisition and computing server 616 (e.g., using SFP and / or a filter cable 620). Some embodiments use a near-infrared (IR) or high-speed IR camera (e.g., a camera capable of acquiring 1000 frames per second) 612 and transfer the images to an image acquisition and computing server (e.g., using a dual camera link full 622). Some embodiments use a high-speed optical camera (e.g., a camera capable of generating a video with a resolution of 1080) to transfer the images to an image acquisition and computing server. In some embodiments, the image acquisition and computing server 616 integrates welding parameters from a weld data acquisition and control system (DACS) 624 and stores the resulting data in a data storage server 618.

[0054] Some embodiments acquire images of various welding areas and include a high-speed IR camera (e.g., FLIR® x6900sc MWIR), a high-speed optical camera (e.g., Blackfly® 0.4 MP / 522 FPS), an operable optical camera, and / or a video camera (e.g., 1080p camera) to maximize prediction capabilities in a single test. Some embodiments use a single camera optimized for the appropriate application. In some embodiments, the high-speed camera has a frame rate from 200 FPS to 1000 FPS, or a frame rate sufficient to acquire features in a very short time, and based on such features, event prediction of welding quality can be performed considering the speed of the torch and differences due to the material. For example, an electron beam welding operation requires a relatively high frame rate to acquire a relatively fast melting pool creation and cooling pattern. For conventional welding with a cooling rate close to 1 second, the camera may reduce the frame rate accordingly, but continue to acquire the operation of the welding pool at a relatively high resolution over a long time. Some embodiments use deep "transfer" learning after the initial data acquisition of basic features to reduce the size and type of the camera required after training of the underlying algorithm. Since some embodiments use existing patterns, a smaller camera (compared to the camera used during training) collects a non-massive dataset for welding determination. Some embodiments use a high-resolution near-infrared camera that is less expensive than FLIR, and a thermal camera and an optical camera (Blackfly 522 FPS operating at about 200 FPS). These cameras are often inexpensive and easily attachable for various applications. Some embodiments are verified using a basic high-speed IR dataset. Some embodiments use a camera mounted coaxially in the welding direction. Some embodiments use data stitching or image processing to restore appropriate dimensions and arrangements for visualization and / or quantification of the data model.

[0055] Some embodiments use multiple computer vision processing methods to predict welding quality characteristics according to the complexity of features of interest. Some embodiments create a statistical model / linear fit that associates the shape of the molten pool with the degree of welding quality using a mathematically defined characterization of the shape of the molten pool. Some embodiments use a deep learning model trained to detect good / bad welding regions based on raw images annotated with inspection results. Some embodiments use various models in combination with 3D visualization of post-weld results. Some embodiments use actual images to obtain the actual boundaries of the shape of the molten pool, and other parameters, welding events, and patterns obtained from successive images of the ongoing welding process.

[0056] For image detection, some embodiments use a high-speed optical camera (e.g., a 200 FPS camera) to acquire electrode splash, changes in the welding pool, and arc patterns, and / or use a 1080p video camera with a laser beam to profile the depth of the weld and the welding progress pattern. Some embodiments use a high-speed infrared camera (e.g., a 1000 FPS camera) to observe the welding process.

[0057] Some embodiments use a thermal camera or an infrared camera to monitor the welding process. Some embodiments acquire the thermal characteristics of both the molten metal and the surrounding solid surface. Some embodiments predict the weld penetration and porosity. Some embodiments utilize full IR and optical images including the solid metal surface and the solidified weld bead. Some embodiments use an optical camera to inspect the weld bead and / or void modeling. Some embodiments use 3D multi-pass weld volume and shape analysis to confirm defect matches. Some embodiments apply deep learning to the image of the weld to identify defects. Some embodiments use thresholding and / or edge detection methods to identify the contour of the weld. Some embodiments use computer vision methods to calculate the area and shape of the actual weld pool and create a statistical model. Some embodiments apply machine learning techniques to the annotated images of the weld bead to classify defects. Referring to FIG. 7J, an example of an infrared image (thermal image) based on some embodiments will be described below.

[0058] Some embodiments provide early warning and enhanced inspection based on detecting welding features leading to defects during or in-situ (as opposed to or in addition to post-weld inspection). In some embodiments, the welding may be stopped upon defect occurrence and the defect corrected in place, saving time wasted in processing and inspection. Some embodiments facilitate easier or less expensive weld repair. For example, weld repairs can be removed by grinding, but if a defect is found after completion and the defect is buried to a depth of several inches, it is time-consuming to remove. Some embodiments notify problem areas to focus NDE on and facilitate accurately identifying problems. Some embodiments facilitate diagnosis and improve feature interpretability. Some embodiments improve weld quality awareness and visualization. Some embodiments include time-event information for tracking any feature to the condition that caused the defect, where the defect cannot be detected in post-processing inspection decoupled from these conditions. Some embodiments facilitate inspection of only defects marked as potential (e.g., without inspecting all features of the product). Some embodiments perform image reconstruction to reinforce incomplete or "fuzzy" NDE and prevent rework. In some embodiments, the techniques described herein serve to replace post-weld inspection techniques with in-weld automated inspection. Some embodiments facilitate making the as-welded condition trackable for future investigations, simulations, or recording of conditions leading to defects. In some embodiments, the welding quality can be initially verified by NDE, but after the initial inspection, no further inspection is required. In some embodiments, images obtained from the ongoing welding process are processed by a deep neural network, which roughly detects and quantifies the characteristics of the weld seam during welding and is trained. Some embodiments automatically acquire and draw attention to quality features that were previously somewhat similar but not explicitly defined or observable. Some embodiments use weighted parameters and probabilities to facilitate automated decision-making and manipulate control variables within acceptable ranges. Some embodiments improve the accuracy, reproducibility, and / or repeatability of welding quality inspection.

[0059] In some embodiments, a camera and data extraction algorithm provide welding characteristic information that is more accurate and reliable than human observation and comparable to the information obtained using NDE, while avoiding the inherent noise, materials, or geometric features of post-processing NDE. Some embodiments use an in-situ system to automatically quantify the defect rate and associate it with changes in the parameters of the automated welding process. Some embodiments reduce the amount of manual inspection and enhance accuracy by assisting human operators in identifying defect areas.

[0060] Some embodiments use high-speed IR time-series mapping. Some embodiments track temperature intensity, melting point, temperature and cooling profiles, and / or the movement of the welding beam. Some embodiments detect features from NIR, high-speed IR, and / or optical cameras. Some embodiments perform a deep learning algorithm for unsupervised feature extraction of the quality features of electron beam welding and associate that information with the CT scan results of the weld. Some embodiments predict the depth of penetration of the weld.

[0061] Some embodiments use a stationary welding machine with a rotating or stationary welding platform. Some embodiments use a high-speed optical camera attached to a tripod placed near the welding platform. In some embodiments, a plastic shield (or similar device) is provided to prevent sparks from damaging the camera lens. Some embodiments use an inert gas welding box. In some embodiments, a computer attached to the camera and / or welding device records data from an IR camera during normal welding operations. In some embodiments, external inspection / external testing is used to identify the location of welding quality defects and high-quality regions and associate them with the acquired data. Some embodiments use an HSIR camera and image processing techniques to image the welding process and predict welding quality problems. Some embodiments predict the depth of penetration of the weld.

[0062] In some embodiments, the welding operation is synchronized with the camera. For example, the camera is attached to a non-fixed tripod. Some embodiments have the camera attached coaxially to the welding arm. Some embodiments zoom in on the camera (e.g., at high resolution) with respect to the welding event or location. Some embodiments always focus on the weld pool and / or cooling locations and provide a reference frame. Some embodiments reduce the complexity of image processing by not considering movement.

[0063] Some embodiments use a thermal cooling gradient. In such cases, the material being welded should be luminescent so that a NIR camera (with a sufficiently high frame rate) can acquire an image of the welding process. For example, some embodiments use a 50MP high-speed optical camera and a NIR camera with filters. Some embodiments use a FLIR high-speed IR camera (e.g., when cooling faster than 1 second). Some embodiments use a small thermal camera depending on the required frame rate. Some handheld cameras are lightweight and can be used by human inspectors. In some embodiments, the images acquired by the camera are analyzed by a computer system (e.g., a system applying machine learning algorithms) to identify welding defects in real-time or during welding. Some embodiments monitor one or more welding parameters including transverse speed, rotation, gas flow, and any control variables associated with a normal or good weld. Illustrations of data preparation

[0064] Some embodiments perform encoded image processing registration, data cleaning, and / or alignment. Some embodiments use lenses selected for an appropriate focal length to achieve these effects. Some embodiments convert the image into a multi-dimensional array representing pixel intensity, color (if required).

[0065] Some embodiments use a convolutional neural network (CNN) or non-linear regression techniques. Some embodiments use autoregressive analysis of time series. Some embodiments train a model against spectra of "good" and "deviant" welds representing many different types. Some embodiments do not require explicitly defining all features.

[0066] In some embodiments, the neural network model learns to predict acceptable welds even when presented with a condition (or image) that was not previously explicitly seen. Some embodiments recognize sub - patterns (e.g., low - level patterns), or patterns based on the complete or overall weld image, and assign, based on probability theory, quality characteristics defined as a standard by the user to new features (i.e., features not seen during training).

[0067] Some embodiments detect features and patterns from the input image, collect data over the entire welding event, assign meaning to the imaged patterns, and perform an automated extraction of engineering features and statistically meaningful interactions with them for optimal characterization, imaging, and quality prediction from the welding process.

[0068] Some embodiments integrate process parameter data from the welding machine, and the process parameter data includes the flow rate, temperature, and pressure of the shielding gas, voltage, current, wire feed speed and temperature (if applicable), pre - heat temperature / inter - pass temperature of the part, and / or the relative speed between the part and the welding torch. Illustrations of applications of computer vision

[0069] In some embodiments, one or more sensors monitor (during the welding process) the shape of the liquid puddle, the deposition of oxides or other contaminants with different emissivities that appear as bright spots, or events and features such as the accumulation, deterioration, and arc wandering of the electrode. Some embodiments use filtering techniques and / or combinations of cameras to highlight features of images such as the electrode, arc, or welding pool.

[0070] In some embodiments, images are sequentially developed and the welding quality in various configurations is displayed. FIG. 7A shows an illustration of an image 700 obtained during a welding process according to some embodiments. FIG. 7A shows bubbles 702 that are formed on the surface of the welding pool and periodically adhere to the side of the welding pool and cool. In some embodiments, an algorithm uses the images to learn good welding patterns, identify anomalies, and / or learn defect patterns to identify low-quality events. For characterization, some features may remain as they appear when the welding pool solidifies.

[0071] FIG. 7B shows an illustration of an image of a welding process according to some embodiments. The image 704 labeled "Start" corresponds to the initial state of the welded material. The next image 706 shows the weld on the left side and an anomaly on the right side. The next image 708 shows a normal state during the progress of the weld on the left side. The next image 710 shows bowing (or bending) on the left side, which is a welding anomaly. The next images 712 and 714 correspond to when both sides are welded and when the center is welded, respectively.

[0072] FIG. 7C shows an illustration of a process 720 for inferring welding quality using a laser profile according to some embodiments. Some embodiments provide a 3D profile 724 of the weld surface and voids using laser light projected onto the surface 722 after welding. Some embodiments perform bead volume analytics 728 based on the video obtained. In FIG. 7C, the red bead 726 has a larger volume and a collapsed shape than the other three colors or bands and shows a state of being detached from the bead. Some embodiments determine the volume and shape of the red bead. Some embodiments inspect physical defects using the mass of the weld. In some embodiments, this type of defect is related to the volume and shape of the analysis digital twin (e.g., the red region 726).

[0073] Figure 7D shows an illustration of a profile progression 730 based on some embodiments. Some embodiments use a machine learning algorithm to train a laser profile and parameterize the profile and alignment regarding quality characteristics during the welding process. In some embodiments, the progression of the laser profile is performed vertically (i.e., the profile plane is perpendicular to the welding surface). In some embodiments, to enhance the learning pattern, the laser profile progression is performed at an angle to the welding surface.

[0074] Figure 7E shows an illustration 732 of an image of an electrode event based on some embodiments. Figure 7F shows an illustration 734 of an image of an arc event of a welding pool based on some embodiments. This example shows the lack of fusion based on some embodiments, and the lack of fusion causes the metal to flow out of the welding pool into the channels (the red regions in the graph of Figure 7C) due to subsurface porosity and subsurface voids. Figure 7G shows an illustration 736 of an image of a defective, ongoing weld based on some embodiments. In particular, the shown defect includes a void (a type of defect) 738. In some embodiments, the micrograph of the weld is used for training a machine learning algorithm. Once the training is complete, the machine learning algorithm uses images of the ongoing weld, such as the shape of the molten pool, the cooling profile, and / or variables that are not explicitly defined (and are obtained from the pattern of the images) to detect the conditions leading to such defects. Figure 7G also shows an illustration of a good joint 740 based on some embodiments. Some embodiments use the illustrations of good joints and voids to train one or more machine learning classifiers to detect welding defects. Some embodiments use appropriate dimensions to display such conditions and / or defects within the digital twin. Figure 7H shows an image 742 that has been unwrapped (or flattened) from a circular-shaped weld based on some embodiments. Some embodiments identify defects related to voids. Some embodiments automatically annotate the image (e.g., label 744-2 and label 744-4) with defects or anomalies. Figure 7I shows the confirmation of a welding defect 746 based on some embodiments.

[0075] Figure 8A shows a welding image 800 of electron beam welding based on some embodiments. In some embodiments, time-series patterns such as welding shape, speed spatter, heat pattern, etc. are acquired by continuous images. Figure 8A also shows an example of an ongoing image 802 including a welding digital twin based on some embodiments, and the welding digital twin includes a heat profile. Some embodiments use this image to form a data layer of the digital twin of the as-welded seam and thereby generate quality features (e.g., defects).

[0076] In some embodiments, the continuous images are aligned with a 3D digital representation of the (predicted) final state of the ongoing welding. Some embodiments show the as-welded boundary with quality features of the final state (the quality features of the final state are either directly observed or predicted with a confidence level higher than a predetermined threshold (e.g., 90% confidence level)) based on the observed welding conditions and / or events.

[0077] Some embodiments acquire the texture of the weld pool, the ripple pattern, and / or the shape of the weld pool. Some embodiments use artificial intelligence to replicate and / or automate the generation of visual indicators to meet the requirements of human inspectors. Some embodiments analyze the cooled state (of the welded product) and associate it with the images acquired during welding to identify the cause of welding defects. Some embodiments use a time-series neural network. Some embodiments use scalar data of measurement parameters. In some embodiments, the camera is calibrated by an optical camera test using resin and sintering processes. Some embodiments generate a digital twin with quality characteristics of interest, and the quality characteristics of interest match CT scans (NDE) and / or DE evaluations (microscopy) in additive manufacturing applications. FIG. 8B shows a CT scan 808 for validating a data model according to some embodiments. FIG. 8B shows a destructive evaluation (DE) hole 810 detected by a microscope, and the microscope shows a conventional destructive evaluation.

[0078] FIG. 9 is an illustration 900 of a process for in-situ inspection of an ongoing welding process according to some embodiments. The weld image 902 is pre-processed (904), classified (906), and the quality characteristics (e.g., welding defects and locations) are automatically extracted (910) and / or visualized before the weld quality is quantified (908). FIG. 9A shows an illustration of the visualization of the number of welding types (or defect types) for each location (class A, class B, and class C) according to some embodiments.

[0079] FIG. 10A is a block diagram of a system 1000 according to some embodiments, which trains one or more regression models (or machine learning models) to identify and / or predict welding defects. Some embodiments obtain a defective welding image 1002 (e.g., a TIG list from NOG-B) and also obtain a welding image without welding defects. In some embodiments, the defective or non-defective welding images are artificially created and used to train a neural network model.

[0080] Some embodiments then pass the welding image to a machine learning model. Instead of using a classification model that directly predicts or identifies welding defects, some embodiments train a regression model to predict welding defects for an ongoing welding process. As will be described below with reference to FIG. 10B, based on the regression model, some embodiments identify or predict welding defects.

[0081] In some embodiments, method 1000 is executed on a computer system, which has one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. This method constructs a regression model 1012 for predicting or identifying welding defects based on some embodiments. The method includes obtaining a plurality of welding images 1002. Each welding image includes either a welding defect or a good weld (i.e., without welding defects). An illustration of welding images was described above with reference to FIGS. 7A-7I according to some embodiments.

[0082] The method includes generating (1004) welding features 1006 by extracting features from the welding image 1002 and integrating one or more welding parameters. The method also includes forming (1008) a feature vector 1010 based on the welding features. The method further includes training (1012) a regression model 1014 (e.g., the machine learning model described above) using the feature vector 1010 to predict or identify welding defects.

[0083] Figure 10B is a block diagram of a system 1020 according to some embodiments, and the system 1020 uses a trained regression model (e.g., the regression model 1014 trained via the process described above with reference to Figure 10A) to facilitate in-situ inspection of a welding process.

[0084] In another aspect, a method (referred to as in-situ inspection of weld quality) for detecting, identifying, and / or visualizing welding defects for an ongoing welding process is provided. This method is executed on a computer system 200, which has one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. This method includes receiving a welding image 1022 from one or more cameras. The method also includes generating (1004) a plurality of welding features based on the welding image 1022 and / or welding parameters as described above with reference to Figure 10A. The method includes forming (1008) a feature vector 1026 v = [v1, v2, …, v n (e.g., as described above with reference to Figure 10A), where the components of the feature vector 1026 include a plurality of features.

[0085] The method further includes predicting or detecting (1028) a welding defect 1030 using a trained classifier (e.g., classifier 1014) based on the feature vector 1026.

[0086] In some embodiments, the method also includes visualizing (e.g., generating a 3D model) (1032) based on the identified welding defect 1030. In some embodiments, the generated 3D or visual model 1034 (e.g., a model for further inspection) confirms (or shows) the welding defect.

[0087] In some embodiments, the method facilitates a user (e.g., a human inspector or operator) visualizing and identifying and / or repairing weld defects and obtaining repaired welded parts (1036).

[0088] The terms used in the description of the invention of the present disclosure are for the purpose of describing particular embodiments only and are not intended to limit the invention. As used in the description of the invention and the appended claims, singular nouns are intended to include the plural as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used in the present disclosure encompasses any and all possible combinations of one or more of the associated and listed items. It will further be understood that the term "comprising" and / or the term "including" as used in the present disclosure identify the presence of the described features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups.

[0089] The foregoing description has been presented for purposes of illustration, with reference to particular embodiments. However, the foregoing exemplary considerations are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many variations and modifications are possible in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, and thus enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular uses contemplated.

Claims

Claim 1 Obtaining a plurality of consecutive images of an ongoing welding process, wherein the plurality of consecutive images have a time series pattern regarding the characteristics of a welding pool, and the characteristics of the welding pool are for the entire welding event of the welding process that forms the area to be welded; Generating a multi-dimensional data input based on the plurality of consecutive images and / or one or more welding process control parameters, Generating the multi-dimensional data input includes: (i) detecting features and patterns from the plurality of consecutive images; (ii) collecting data over the entire welding event; (iii) assigning meaning to the patterns; (iv) integrating one or more welding process control parameter data; (v) extracting statistically significant features; The pattern includes at least one of the shape of the welding pool, the size of the welding pool, the contour of the welding pool, the depth of the welding pool, the temperature gradient in the welding pool, the change with the passage of time in the welding pool, and the uniformity in the welding pool; Collecting data over the entire welding event includes geometrically aligning the plurality of consecutive images using temporal cross-correlation; Generating the multi-dimensional data input further includes converting the temporal trend of the images into a stationary signal by temporally differentiating the plurality of consecutive images; Generating the multi-dimensional data input; Generating a defect probability and analysis information by applying one or more computer vision techniques to the multi-dimensional data input, The analysis information includes predictive insights regarding the quality characteristics of the ongoing welding process, The quality characteristics include the type and location of welding defects, The one or more computer vision techniques include applying a convolutional neural network using 3D convolution to a batch of the plurality of consecutive images that are consecutive and have a delay; The 3D convolution is applied to at least one of pixel position, intensity, and color / spectral band; Generating the defect probability and the analysis information; Generating 3D visualization of one or more as-welded areas based on the analysis information and the plurality of consecutive images, wherein the 3D visualization displays quality characteristics for virtual inspection and / or quality characteristics for determining weld quality, and generating the 3D visualization; A method for in-situ inspection of weld quality, including this.

2. The method according to claim 1, wherein the one or more computer vision techniques include one or more trained machine learning algorithms trained to identify anomalies or defects in the ongoing welding process based on the plurality of consecutive images.

3. The method according to claim 2, wherein the one or more trained machine learning algorithms include one or more trained unsupervised anomaly detection algorithms trained by images of welded joints that meet quality criteria to identify weld defects based on the plurality of consecutive images.

4. The method according to claim 3, wherein the one or more trained machine learning algorithms include one or more trained supervised anomaly detection algorithms trained by images of weld defects classified as not meeting quality criteria to identify weld defects based on the plurality of consecutive images.

5. The method according to claim 2, wherein the one or more trained machine learning algorithms are trained by images of weld defects classified as not meeting quality criteria to identify weld defects based on the plurality of consecutive images.

6. The multi-dimensional data input includes a multi-dimensional array representing pixel intensity and color. The one or more trained machine learning algorithms include a trained convolutional neural network (CNN). The trained convolutional neural network is trained to identify the boundaries, speed, spatter, rate of change, and / or welding parameters of the shape of the weld pool based on the multi-dimensional array, and to determine weld quality, defects, and / or one or more characteristics in the ongoing welding process. The method according to any one of claims 2 to 5.

7. The method according to claim 6, wherein the convolutional neural network identifies the boundaries of the shape of the weld pool by using threshold processing or edge detection methods to identify the contours.

8. Welding process control parameters include the flow rate, temperature, and pressure of the shielding gas, and Voltage, current, wire feed speed, and optionally temperature, and the preheat temperature / inter-pass temperature of the component, the relative speed between the component and the welding torch, The method according to any one of claims 1 to 7, comprising one or more of the above.

9. In response to a determination that the quality characteristics of the ongoing welding process do not meet a predetermined quality standard, stopping the ongoing welding process, generating a warning for one or more events of the ongoing welding process based on the analysis information, The method according to any one of claims 1 to 8, further comprising the above.

10. The virtual inspection of the as-welded area is used to determine the welding quality of the as-welded area, according to the method of claim 1.

11. The pattern further includes at least one of an intensity pattern in the weld pool, spatter in the weld pool, and alignment in the weld pool, according to the method of claim 1.

Images