Systems and methods for multi-spot beam tracking

Abstract

A method for tracking multiple beams within a weld pool is provided. The method includes acquiring an input video feed including multiple frames with multiple beams for the weld pool. The method also includes creating, for each frame of the multiple frames of the input video feed, a gradient mask that identifies one or more pixels in the respective frame related to the weld pool. The method also includes detecting and filtering contiguous regions based on the gradient mask to obtain a pixel mask for each contour of the weld pool. The method also includes identifying locations of the one or more beams within the weld pool based on the pixel mask for each contour.

Description

[Technical Field]

[0001] Related application data This application is based on and claims the benefit of priority under 35 U.S.C. § 119 of U.S. Provisional Application No. 63 / 442,587, filed February 1, 2023, the entire contents of which are incorporated herein by reference.

[0002] STATEMENT OF FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT The invention described herein was made in the performance of work under DOE Cooperative Agreement No. DE-NE0008744. The Government has certain rights in this invention.

[0003] FIELD The disclosed implementations relate generally to manufacturing, and more particularly to systems and methods for multi-spot beam tracking. [Background technology]

[0004] A key data source for all welding-based operations (e.g., powder bed fusion additive manufacturing) is the video feed of the weld pool. The video feed of the weld pool allows for analysis of the weld geometry and thermal profile. The video feed of the weld pool can be used for material property prediction, defect detection, and other purposes. For this reason, some laser powder bed fusion machines include a thermal camera that coaxially tracks the welding beam using mirrors that also control the beam positioning. However, not all laser machines have this capability. For example, in electron beam systems, this capability is physically impossible because the beam is not controlled by mirrors. Machines without a coaxial camera may be equipped with a camera system that has a fixed viewpoint of the build area. For example, electron-beam melting (EBM) systems may be retrofitted with a high-speed infrared camera to record build data. To effectively use the build data to characterize an operation, it is necessary to track the location of the weld pool so that it can be related to the component geometry. Tracking a single beam is a trivial computational problem, as it is usually the brightest or hottest point in the image, but some laser systems and most EBM systems can hold multiple weld pools simultaneously. Summary of the Invention [Means for solving the problem]

[0005] Thus, there is a need to detect and track the locations of any number of weld pools or beams within a live video feed or a stored video file. The techniques described herein may be performed in real time, allowing data analysis and targeted video recording to be performed on high-speed feeds that would otherwise generate too much data. Furthermore, these techniques can be used to effectively track beams in systems that use pulsed or oscillating beam locations. The methods described herein can be used to identify generated beam paths in systems that use black-box path generation. These methods are useful for automatically detecting beam intersections, which correlate with defect formation. Additionally, these techniques can be used to track beams that are close to each other. These techniques provide the ability to track any number of beams in a variety of complex situations. In addition to weld pools and beams, these techniques can be used to track the specific movement paths of other moving objects, such as particles, sparks, slag, and any high-contrast or bright objects that may not be distinctive enough to track with conventional feature detectors. Additionally, the techniques described herein may be directly integrated into control systems or other applications for welding and / or manufacturing.

[0006] According to some implementations, the method is performed on a computing system. Typically, the computing system includes a single computer or workstation, or multiple computers, each having one or more CPU and / or GPU processors and memory. The method generally does not require a computing cluster or supercomputer. The method can be used to track multiple beams within a weld pool.

[0007] The method includes acquiring an input video feed including a plurality of frames having a plurality of beams for the weld pool. The method also includes creating, for each frame of the plurality of frames of the input video feed, a gradient mask that identifies one or more pixels in the respective frame related to the weld pool. The method also includes detecting and filtering contiguous regions based on the gradient mask to obtain a pixel mask for each contour of the weld pool. The method also includes identifying locations of the one or more beams within the weld pool based on the pixel mask for each contour.

[0008] In some implementations, identifying the one or more pixels includes calculating, for each pixel in each frame, a time-based derivative of the intensity of each pixel based on the respective frame of the video feed and its predecessor frame, and determining a potential location of the weld pool based on the time-based derivative of the intensity of each pixel.

[0009] In some implementations, determining the potential location of the weld pool includes calculating at least one of: (i) the relative intensity of each pixel relative to a background image; (ii) the emission intensity of the material used for welding in the near-infrared (NIR) spectrum and the infrared (IR) spectrum; or (iii) a sudden drop in intensity.

[0010] In some implementations, determining potential locations of the weld pool includes thresholding the time-based derivative of each pixel with a high-low pass filter to create a gradient mask, thereby marking pixels that either increase or decrease in value sharply as potential locations of the weld pool.

[0011] In some implementations, a high-low pass filter is applied to the time-based derivative and includes two thresholds: (i) a low value that is a negative number, below which a delta value represents a phase change from powder to liquid metal, and (ii) a high value that is a positive number, above which either heat is increasing or the beams are crossing on the pixel.

[0012] In some implementations, the method further includes defining a respective threshold value for each weld type.

[0013] In some implementations, the method further includes using a morphological close filter on the gradient mask to fill in any gaps between isolated pixels surrounding possible melt pool locations.

[0014] In some implementations, obtaining a pixel mask for each contour of the weld pool includes detecting the contours based on a gradient mask, filtering the contours by size with a high-pass filter to remove all regions that are too small to plausibly be the weld pool, and outputting a pixel mask for each remaining contour.

[0015] In some implementations, identifying the location of one or more beams within each weld pool includes identifying peak intensities by calculating a local maximum value within a pixel mask of each contour; identifying missing beams for each contour by determining whether the peak intensity is below a predetermined range; applying a high-pass threshold to a surrounding area based on the peak intensity to determine a spot area; and calculating the coordinates of the beam location based on the average value or centroid of the spot area.

[0016] In some implementations, the step of locating one or more beams within each weld pool further includes (i) measuring the overall size and aspect ratio of the weld pool, and (ii) responsive to determining that the weld pool has a significantly increased size or a slightly increased size and an unusual aspect ratio, locating the multiple beams within the weld pool by detecting a bimodal peak within the high intensity range.

[0017] In another aspect, a method for tracking multiple beams in a weld pool is provided. According to some implementations, the method is executed in a computing system. Typically, the computing system includes a single computer or workstation, or multiple computers, each having one or more CPU and / or GPU processors and memory. The method generally does not require a computing cluster or supercomputer. The method can be used to track multiple beams in a weld pool.

[0018] The method includes detecting, for each frame of the video feed, one or more events, each event corresponding to a location of a respective beam within the melt pool in the respective frame. The method also includes generating a queue of match scores between the one or more events and the plurality of beam trackers. Each match score represents a degree of match between a respective event of the one or more events and a respective beam tracker of the plurality of beam trackers. Each beam tracker is associated with a list of detected event objects. Each detected event object is associated with a beam of the one or more beams. The method also includes processing the queue of match scores to identify a respective beam tracker of the plurality of beam trackers to pair with each event of the one or more events. The method also includes processing unpaired events of the one or more events to associate the unpaired events with (i) a corresponding tracker of the plurality of beam trackers, or (ii) a new tracker, including adding the new tracker to the plurality of beam trackers.

[0019] In some implementations, detecting the one or more events includes identifying the weld pool by detecting and filtering a contiguous area in each frame, storing a bounding box of the weld pool in a new detection event object for an event of the one or more events, detecting a peak intensity and identifying a location of the beam in the weld pool based on high-pass thresholding a surrounding area at the peak intensity, and storing coordinates of the beam location in the new detection event object.

[0020] In some implementations, the step of detecting one or more events further includes determining whether the molten pool includes multiple beam locations by measuring the overall size and aspect ratio of the molten pool, and generating a second detection event object for a second beam location within the molten pool in response to determining that the molten pool includes multiple beams and in response to determining that the peak intensity has a bimodal peak.

[0021] In some implementations, generating a queue of match scores includes (i) determining the position P of each event E , (ii) the last known location P of each beam tracker T , (iii) the number of time steps since the last detection in each beam tracker, Δ T , (iv) the predicted next location P of each beam tracker P , (v) average global beam velocity per timestep μ v , and (vi) a user-defined maximum speed V max and adding the match score to a queue of match scores.

[0022] In some implementations, generating a queue of match scores comprises: E and P T The distance between max * Δ T responsive to determining that the match score exceeds the threshold, forgoing adding the match score to the queue.

[0023] In some implementations, the match score corresponds to the magnitude of acceleration / deceleration normalized to the average velocity, hereafter:

number

number

number

[0024] In some implementations, processing the queue of match scores includes determining, based on the queue of match scores, whether each event of the one or more events can be paired with each beam tracker of the multiple beam trackers. An event can be paired with a beam tracker only if the corresponding match score of that event in the queue of match scores exceeds a predetermined threshold. The method also includes, for each beam tracker of the multiple beam trackers, removing the respective beam tracker from the queue when the respective beam tracker cannot be paired with any event of the one or more events. When each beam tracker can be paired with two or more events of the one or more events, the method sorts the pairs and selects the best-matching event based on the match score of the pair. When each beam tracker can be paired with only one event of the one or more events and only one event can be paired with two or more beam trackers, the method also includes adding the paired beam tracker to an intersection list of only one event and updating the intersection time step of the paired beam tracker. When (i) each beam tracker corresponds to a best match of only one event, or (ii) only one event can be paired with only the respective beam tracker, the method includes only one event in the list of detected event objects for each beam tracker, removes from the queue all pairs that include either only one event or the respective beam tracker, and removes the respective beam tracker from the queue.

[0025] In some implementations, the step of processing unpaired events includes determining a crossing time threshold CTT, which is the maximum amount of time a beam can take to separate or re-emerge after crossing another beam, using the formula

number

[0026] In some implementations, the step of processing unpaired events involves checking all events for all beam trackers with timestamps within the last CTT time step for entries in the intersection list while checking for the following two values: (a) if on a machine with a pulsed beam or a beam that cycles rapidly between locations;

number

[0027] In some implementations, the method further includes extracting a molten pool image for each beam tracker of the plurality of beam trackers that is active in each video frame of the video feed.

[0028] In some implementations, each beam tracker further includes (i) a unique identifier, (ii) a video output file or object for the video being extracted, (iii) a cache list or vector of images, and (iv) an integer indicating the last intersection timestep.

[0029] In some implementations, each detection event object includes (i) the X / Y coordinates of the beam, (ii) the frame number or time step, (iii) the bounding box and / or pixel mask of the detected melt pool area, and (iv) a list of tracker identifiers in the case of intersection events.

[0030] In some implementations, a computing system includes one or more computers, each of which includes one or more processors and a memory. The memory stores one or more programs configured for execution by the one or more processors. The one or more programs include instructions for performing any of the methods described herein.

[0031] In some implementations, a non-transitory computer-readable storage medium stores one or more programs configured for execution by a computing system having one or more computers, each computer having one or more processors and memory, the one or more programs including instructions for performing any of the methods described herein.

[0032] Thus, a method and system for efficiently tracking multiple spot beams for manufacturing is disclosed.

[0033] For a better understanding of the disclosed systems and methods, as well as additional systems and methods, reference should be made to the Detailed Description below in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout the figures: [Brief explanation of the drawings]

[0034] [Figure 1] FIG. 10 is a diagram of an exemplary process for multi-spot beam tracking according to some implementations. [Figure 2] 1 is a flowchart of an exemplary method for matching a beam spot with a tracker, according to some implementations. [Figure 3A] A diagram showing an example color-mapped beam path showing detected beams (shading corresponds to the color of the original image) according to some implementations. [Figure 3B]FIG. 1 illustrates an exemplary color mapping of beam paths according to some embodiments (shading corresponds to colors in the original image). [Figure 4] FIG. 1 is a block diagram illustrating a computing device according to some implementations. [Figure 5] 1 is a flow diagram of an exemplary method for tracking multiple beams in a molten pool, according to some implementations. [Figure 6] 10 is a flow diagram of another example method for tracking multiple beams in a molten pool, according to some implementations. DETAILED DESCRIPTION OF THE INVENTION

[0035] Reference will be made hereinafter to implementations, examples of which are illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details.

[0036] Multiple weld pools create challenges. For example, simply finding the brightest point in an image is not enough to find the beam. Alternative methods must be used. Furthermore, electron beam systems can have any number of weld pools or beams. This number can be 20 or more. The number of weld pools present can change over time. When multiple beams are present, the beams may approach or even intersect each other, leading to tracking issues for most motion tracking algorithms. Weld consistency is most variable at the beginning and end of a weld. For this reason, some implementations identify the start and stop locations of each weld over time. Some implementations track the instantaneous location of any beam or weld pool in each image, log the beam's directional velocity and acceleration, identify beam intersections, and / or retroactively interpolate beam positions during single or combined weld pools. Some implementations track new beam formations and / or log start and stop points. Some implementations extract video cropped around each beam, thereby enabling weld analysis methods designed for coaxial feeds on machines that do not have coaxial cameras.

[0037] FIG. 1 is a diagram of an example process 100 for multi-spot beam tracking according to some implementations. Some implementations process input video frames (102), detect melt pools (104), and / or identify beam spots (106). Some implementations associate beam spots with trackers (108). Some implementations handle edge cases and / or beam intersections (110). Some implementations clean up stalled beams (112). Some implementations write out video and tracking files (114). Some implementations iterate over frames of the video feed (116). These steps are described in more detail below.

[0038] The techniques described herein can be used to track multiple beams and / or weld pools within a single video. In some implementations, the beam location is tracked frame-by-frame using a beam tracker data structure. The beam data structure may include: (i) a unique identifier; (ii) a video output file or object (if video is being extracted); (iii) a cache list / vector of images; (iv) a list / vector of detection event objects; (v) an integer representing the last "intersection time step," which is by default the minimum value (e.g., the smallest possible integer that can be represented in the system); and / or (vi) a detection event object containing the following data: the beam's X / Y position, the frame number / time step, the bounding box and / or pixel mask of the detected weld pool area, and / or a list of tracker identifiers in the case of intersection events.

[0039] In some implementations, beam tracker objects are stored in one or more vectors / lists (e.g., an active list and a completed list). Initially, these vectors / lists may be empty. For each frame of the video / feed, some implementations (i) create an empty list / vector of detected events for the current frame and (ii) identify pixels that may contain the melt pool. Rather than using a threshold on the image itself and looking for a maximum, some implementations use two consecutive frames of the video feed to identify the time-based derivative (or gradient) of the intensity of each pixel. This may be a simple pixel-by-pixel subtraction.

[0040]

number

[0041] In some implementations, the potential location of the weld pool is determined based on observing pixels on or surrounding the pool. The beam is typically moving and brighter than the background image, and therefore likely to have a high positive time delta of intensity. The beam is imparting heat to the surrounding material. If the material does not undergo a phase change from solid to liquid, the material's luminescence intensity is likely to increase. This is especially true in the NIR and IR spectra. If the material does undergo a phase change, the luminescence decreases. Liquid metal typically emits less light in the visible, NIR, and IR spectra than the heated solid metal around it. This is likely to result in a sudden drop in intensity.

[0042] Some implementations threshold the time derivative of each pixel with a high-low pass filter to create a gradient mask. Some implementations define a threshold for each pixel. In some implementations, pixels that either increase or decrease rapidly in value are marked as potential weld pool locations. This threshold need not be particularly conservative. Some implementations use a morphological closing filter on the gradient mask to fill in all gaps between isolated pixels surrounding potential weld pool locations. Some implementations detect and / or filter contiguous regions. Some implementations apply a contour detection algorithm to the gradient mask. Some implementations filter contours by size with a high-pass filter to remove regions that may be too small to be plausible weld pool locations. Some implementations store the weld pool bounding box in a new detection event object and add the new detection event object to the event list for the current frame. Some implementations output a pixel mask for each remaining contour. Some implementations identify the location of the beam within the weld pool. This may be done by finding a local maximum within each contour's pixel mask.

[0043] For electron beam systems, the beam may not be visible in every frame as it moves quickly between different weld pools. Whether the beam has stopped may be determined in different steps. Assume the beam is still active. Some implementations check if the peak intensity is within the range normally expected for the beam (e.g., the top 1% of the intensity range in the image). If the intensity is lower than expected, some implementations mark the beam as a "missing beam" and leave the X / Y location empty. Once a peak is detected, some implementations conservatively high-pass threshold the surrounding area based on the peak value to determine the spot area. In some implementations, the X / Y location of the beam is the mean / centroid of the spot area. Some implementations store the X / Y location of the peak (if applicable) in this weld pool detection event object.

[0044] Because the beams may intersect with each other, there may be multiple beam locations within the same weld pool. Whether this is the case can be easily determined by measuring the overall size and aspect ratio of the weld pool. This is because doubling the welding thermal input rapidly expands the width and potentially the length of the pool. If the weld pool has a significantly increased size, or a slightly increased size and an unusual aspect ratio, some implementations check for a bimodal peak within the high-intensity range. If so, some implementations create another detection event for the second beam.

[0045] Some implementations create a queue of match scores to evaluate the compatibility between detected events and existing trackers. For each combination of tracker and detected event, some implementations calculate a match score, which is the following value: (i) the position of the detected event (P E ), (ii) the tracker's last known location (P T ), (iii) the number of time steps since the last detection in the tracker (ΔT), and (iv) the predicted next location of the tracker (P P) which is typically the last position of the tracker plus the delta between the last two detected positions of the tracker multiplied by the number of time steps since the last detection. In applications with low resolution, slow beam movement speeds, or high frame rates, this delta may need to be calculated based on multiple previous values. If the tracker is new (has only one value), this value may be ignored. The determination is based on (v) the average global beam velocity per time step (μ V ), and / or (vi) a user-defined maximum speed (V max ) is also used.

[0046] Some implementations pre-filter matches by distance: T , P E ) > (V max * Do not create records for any matches where ΔT) applies.

[0047] In some implementations, the match score is an ordered structure that includes one or more of the following values:

number

number

number

[0048] Some implementations add each match score object to a queue.

[0049] In some implementations, matches are compared to each other using the following method: Each metric is compared in turn. This structure is prioritized to weight matches with known speed and direction of beam movement, and if not, to favor beam movement closest to the average beam speed. If both match score metrics are within a defined tolerance (e.g., 10% of the total value), the next metric is evaluated. Otherwise, the system favors the match with the lowest score. If all of these metrics fail to produce a clear winner, the system runs the same test without the tolerance. If this test fails, the system arbitrarily picks one (a perfect match) since the distinction between the two beams / molten pools is inappropriate at this point (and they likely intersect).

[0050] FIG. 2 is a flow diagram of an example method 200 for matching beam spots with trackers, according to some implementations. Some implementations process a queue of matching scores (sometimes called a processing queue). Some implementations use an iterative algorithm for matching. The steps shown in FIG. 2 may be performed for each tracker in the processing queue until the processing queue is empty. Assume there is an input beam tracker 202. Some implementations calculate (206) the number of paired detection events. If there are no paired detection events, the algorithm stops after removing (204) the tracker from the processing queue. If there are multiple paired detection events, some implementations sort (208) the pairs and select the best-matching event. Some implementations then calculate (210) the number of trackers paired with the event. If there is only one such tracker, some implementations add (216) the event to the tracker's detection record. If there are multiple trackers paired with the event, some implementations add all paired trackers to the event's intersection list (212), if they are not already present, and update the tracker's intersection timestep. Some implementations determine (218) whether the current tracker is the best match for the event. If so, or if there is only one tracker paired with the event, the event is added to the tracker's detection record (216). If not, the tracker is moved to the end of the processing queue, and the algorithm skips (220) to the next tracker (in the queue). After adding the event to the tracker's detection record, the algorithm removes (214) all pairs containing either the event or the tracker from the list, after which the tracker is removed from the processing queue (204), and the algorithm stops.

[0051] Some implementations use a Hungarian combinatorial optimization algorithm to process the queue. Some implementations process vector cost values. In practice, this may not be necessary for beam matching, as situations where there are more than three possible matches are rare. Since the velocity of the beam is unlikely to change, the best match between the beam and the predicted value is generally the correct match. Compared to this method using the Hungarian combinatorial optimization algorithm, the method described above with reference to FIG. 2 maximizes the number of best matches.

[0052] Some implementations handle unpaired events as described herein. Each unpaired event is either associated with an existing tracker or is the start of a new tracker. An intersection time threshold is defined as the maximum amount of time after a first beam intersects with another beam before it should split or reappear (the first beam is likely to only be detected and associated with one tracker). This threshold is calculated based on the following formula:

[0053]

number

[0054] In some implementations, the threshold may also be manually set to some other value (e.g., a value smaller than that given by the above formula if the beams are easily separated from each other within the same molten pool).

[0055] Some implementations check for detection events on all trackers with a timestamp within the last CTT timestep for entries in the intersection list. Some implementations create an empty pairing list. For each event with a list, some implementations check dist(P T , P E ) < VMAX * ΔT. If so, some implementations perform the following steps for each tracker in the list (except the tracker containing the intersection event): Some implementations create a what-if score for the tracker as if the intersection event had been inserted into the tracker's history. Some implementations calculate dist(P T , P E ) < V T * Check whether ΔT and V T is a rate based on the tracker's previous event and the intersection event. If not, some implementations skip this tracker. If this tracker already has a subsequent event, some implementations check whether the subsequent event's match score is better than the intersection event. If so, this tracker is skipped. If not, or if there are no subsequent events, the tracker / new event is added to the pairing list. If the pairing list is not empty, some implementations select the tracker with the best match score. If the match already has any associated events after it is found, some implementations move these events to a new tracker and add that tracker to the active list. Some implementations duplicate the intersection event in this tracker's history. Some implementations add unpaired events to the best-matching tracker. In some implementations, a new tracker is created with any unpaired events as its first event.

[0056] Some implementations clean up the active tracker list as described herein. Beams that have not been detected within a certain period of time are moved from the active list to the completed list (because they have been turned off / stopped). This value is called the inactivity threshold (IT). This upper threshold has two values: (i) if on a machine with a pulsed beam or a beam that cycles rapidly between locations;

[0057]

number

[0058] Some implementations extract weld pool images for each active beam tracker as described herein. A rolling buffer of at least the full IT image is kept in case a crossing update or other reason requires a beam position correction. If the tracker's crossing time step is less than the CTT time step before the current time step, some implementations do nothing. If the current time step is exactly the CTT time step after the crossing time step, some implementations process all of the images from the previous CTT frame before the current frame. If the tracker is terminated, some implementations process all frames since the last event. The region of interest (ROI) is a window of a predetermined size centered on the beam. For each time step being processed, some implementations perform the following steps: If there are no events in this time step, if the tracker is terminated, the center location is the last known beam location from the last event; if there is an event after this time step, linear interpolation between the previous and next events is used to determine the beam position / center location; otherwise, skip to the next tracker (this beam finishes processing). If not, some implementations take the beam position from the current event as the center location and / or crop the image to a ROI around the center location. Some implementations write the cropped image to an output video file or object.

[0059] 3A and 3B show example beam paths from high-speed infrared video, according to some implementations. Fig. 3A shows an example color-mapped beam path 300 (shown here by shading) showing dozens of detected beams. Each color corresponds to the path of a beam. Fig. 3B shows an example color mapping 302. Start or stop points are shown in purple, and points where the beams intersect are shown in red.

[0060] 4 is a block diagram illustrating a computing device 400 according to some implementations. Various examples of computing device 400 include high performance clusters of servers (HPC), supercomputers, desktop computers, cloud servers, and other computing devices. Computing device 400 typically includes one or more processing units / cores (CPUs and / or GPUs) 402 for executing modules, programs, and / or instructions stored in memory 414 to thereby perform processing operations, one or more network or other communication interfaces 404, and one or more communication buses 412 for interconnecting the memory 414 and these components. Communication bus 412 may include circuitry for interconnecting and controlling communication between system components.

[0061] The computing device 400 may include a user interface 406 that includes a display device 408 and one or more input devices or mechanisms 410. In some implementations, the input device includes a high-speed infrared camera (e.g., a high-speed infrared camera in an EBM system) that may be used to acquire video frames of the welding process (sometimes referred to as a video feed or an input video feed). In some implementations, the input device / mechanism includes a keyboard. In some implementations, the input device / mechanism includes a "soft" keyboard that is optionally displayed on the display device 408, allowing a user to "press keys" that appear on the display device 408. In some implementations, the display device 408 and the input device or mechanism 410 include a touchscreen display (also referred to as a touch-sensitive display).

[0062] In some implementations, memory 414 includes high-speed random-access memory such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, memory 414 includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some implementations, memory 414 includes one or more storage devices located remotely from GPU / CPU 402. Memory 414, or alternatively, a non-volatile memory device within memory 414, includes a non-transitory computer-readable storage medium. In some implementations, memory 414, or the computer-readable storage medium of memory 414, stores the following programs, modules, and data structures, or a subset thereof: an operating system 416, which includes procedures for handling various basic system services and performing hardware-dependent tasks; a communications module 418 used to connect the computing device 400 to other computers and devices via one or more communications network interfaces 404 (wired or wireless) and one or more communications networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.; an optional data visualization application or module 420 for displaying visualizations of the model and / or beamforming; an imaging module 422 for acquiring images 424 (sometimes referred to as frames or image frames), for example, the imaging module 422 may acquire frames of a video feed including multiple beams in the weld pool; a gradient mask generation module 426 for generating (or creating) a gradient mask 428 that identifies pixels in the frame related to the melt pool; a pixel mask generation module 430 for detecting and filtering contiguous regions based on the gradient mask to obtain a pixel mask 432; a beam location determination module 434 that determines a location 436 of the beam within the weld pool based on the pixel mask; and / or an optional beam tracking module 438 for (i) detecting events 440 in video frames corresponding to the location of the beam in the melt pool; (ii) generating a queue of match scores 442 between events and beam trackers 444, each match score representing the degree of match between a respective event and a respective beam tracker (each beam tracker may be associated with a list of detected event objects 446, and each detected event object may be associated with a beam); (iii) processing the queue of match scores to identify each beam tracker to pair with the event; and / or (iv) processing unpaired events to associate such events with the corresponding tracker or a new tracker.

[0063] Each of the above-identified executable modules, applications, or sets of procedures may be stored in one or more of the aforementioned memory devices and correspond to sets of instructions for performing the functions described above. The above-identified modules or programs (i.e., sets of instructions) are not necessarily implemented as separate software programs, procedures, or modules. Thus, various subsets of these modules may be combined or otherwise rearranged in different implementations. In some implementations, memory 414 stores a subset of the above-identified modules and data structures. Additionally, memory 414 may store additional modules or data structures not described above. The operation of each module and properties of the data structures shown in FIG. 4 are further described below according to some implementations.

[0064] 4 illustrates a computing device 400, which is intended as a functional description of various features that may be present rather than as a structural schematic of the implementations described herein. In practice, those skilled in the art will recognize that items shown separately may be combined and some items may be separated.

[0065] Details of the modules and data structures shown in FIG. 4 are described above with reference to FIGS. 1-3B and, according to some implementations, are further described below with reference to the flow diagrams shown in FIGS. 5 and 6.

[0066] 5 is a flow diagram of an example method 500 for tracking multiple beams in a weld pool, according to some implementations. The method is performed in an electronic device (e.g., computing device 400) having one or more processors (e.g., CPU / GPU 402) and memory (e.g., memory 414) that stores one or more programs configured for execution by the one or more processors.

[0067] The method includes acquiring 502 an input video feed including a plurality of frames having a plurality of beams for a weld pool. For example, an imaging module 422 acquires an image 424.

[0068] The method also includes creating (504), for each frame of the multiple frames of the input video feed, a gradient mask that identifies one or more pixels in the respective frame related to the weld pool. For example, the gradient mask generation module 426 generates the gradient mask 428. In some implementations, identifying the one or more pixels includes calculating, for each pixel in the respective frame, a time-based derivative of the intensity of the respective pixel based on the respective frame and the preceding frame of the video feed, and determining a potential location of the weld pool based on the time-based derivative of the intensity of each pixel. In some implementations, determining the potential location of the weld pool includes calculating at least one of: (i) the relative intensity of each pixel relative to a background image; (ii) the emission intensity of the material used for welding in the near-infrared (NIR) spectrum and the infrared (IR) spectrum; or (iii) a drop in intensity. The relative intensity typically includes a high positive time delta of intensity. The beam is typically moving and brighter than the background image, thus resulting in a high positive time delta of intensity. The beam typically imparts heat to the surrounding material due to the emission intensity. When a material does not undergo a phase change from solid to liquid, the luminescence intensity of the material typically increases. This is especially true in the NIR and IR spectra. When a material undergoes a phase change, the luminescence typically decreases. Liquid metal typically emits less luminescence in the visible, NIR, and IR spectra than the surrounding superheated solid metal. This often results in a sudden drop in intensity. In some implementations, determining potential locations of a weld pool involves thresholding the time-based derivative of each pixel with a high-low pass filter to create a gradient mask, thereby marking pixels that either increase or decrease in value as potential locations of a weld pool. In some implementations, the high-low pass filter is applied to the time-based derivative.The high-low pass filter includes two thresholds: (i) a low value, a negative number below which delta values ​​represent a phase change from powder to liquid metal; and (ii) a high value, a positive number above which either heat is increased or the beam is crossing the pixel. The high-low pass filter excludes the band between the two thresholds (values ​​closer to zero delta) and selects either the high or low end, but not the middle. These thresholds need not be particularly conservative. Exemplary values ​​are 25 and 75 for the low and high thresholds, respectively. In some implementations, the method further defines respective thresholds for each weld type. Some implementations provide a user interface that displays a single colorized delta image and sliders for each setting to highlight appropriate areas for the user to select thresholds. In some implementations, the method further includes using a morphological closing filter on the gradient mask to fill gaps between isolated pixels surrounding potential weld pool locations.

[0069] The method also includes detecting and filtering contiguous regions based on the gradient mask to obtain a pixel mask for each contour of the weld pool (506). For example, pixel mask generation module 430 generates pixel mask 432. In some implementations, obtaining the pixel mask for each contour of the weld pool includes detecting contours based on the gradient mask (e.g., applying a contour detection algorithm to the gradient mask), filtering the contours by size with a high-pass filter to remove all regions that are too small to plausibly be a weld pool (e.g., this may be application specific, or spots less than 100 pixels may be considered too small to be a weld pool), and outputting a pixel mask for each remaining contour.

[0070] The method also includes identifying (508) one or more beam locations within the weld pool based on each contour's pixel mask. For example, the beam location module 434 identifies beam locations 436. In some implementations, identifying one or more beam locations within each weld pool includes identifying peak intensities by calculating a local maximum within each contour's pixel mask; identifying missing beams for each contour by determining whether the peak intensity is below a predetermined range (typically the top 1% of the intensity range in the image); applying a high-pass threshold to the surrounding area based on the peak intensity to determine a spot area; and calculating beam location coordinates based on the average or centroid of the spot area. In some implementations, identifying one or more beam locations within each weld pool further includes (i) measuring the overall size and aspect ratio of the weld pool; and (ii) identifying multiple beam locations within the weld pool by detecting bimodal peaks within the high-intensity range when the weld pool has a significantly increased size or a moderately increased size and an unusual aspect ratio. Because beams may intersect with each other, multiple beam locations may exist within the same weld pool. Some implementations determine whether this is likely to be the case by measuring the overall size and aspect ratio of the weld pool. Doubling the heat input of the weld typically results in a rapid expansion of the pool's width and potential length.

[0071] 6 is a flow diagram of an example method 600 for tracking multiple beams in a weld pool, according to some implementations. The method is performed in an electronic device (e.g., computing device 400) having one or more processors (e.g., CPU / GPU 402) and a memory (e.g., memory 414) that stores one or more programs configured for execution by the one or more processors.

[0072] The method includes acquiring 602 an input video feed including a plurality of frames having a plurality of beams for a weld pool. For example, the imaging module 422 acquires the image 424.

[0073] The method includes performing (604) a sequence of steps for each frame of the video feed. The sequence of steps may be performed by the beam tracking module 438. The method includes detecting (606) one or more events 440, each event corresponding to a respective beam location within the weld pool in a respective frame. In some implementations, detecting the one or more events includes identifying the weld pool by detecting and filtering contiguous areas within each frame, storing a bounding box of the weld pool in a new detection event object for the one or more events, detecting a peak intensity and identifying a beam location within the weld pool based on high-pass thresholding a surrounding area at the peak intensity, and storing the beam location coordinates in the new detection event object. In some implementations, detecting the one or more events further includes determining whether the weld pool has multiple beam locations by measuring the overall size and aspect ratio of the weld pool, and generating a second detection event object for a second beam location within the weld pool when the weld pool includes multiple beams and the peak intensity has a bimodal peak.

[0074] The method also includes generating (608) a queue of match scores 442 between the one or more events and the multiple beam trackers. Each match score represents a degree of match between a respective event of the one or more events and a respective beam tracker of the multiple beam trackers. Each beam tracker is associated with a list of detection event objects 446. Each detection event object is associated with a beam of the one or more beams. In some implementations, each beam tracker further includes (i) a unique identifier, (ii) a video output file or object for the video being extracted, (iii) a cache list or vector of images, and (iv) an integer indicating the last intersection time step. In some implementations, each detection event object includes (i) the X / Y coordinates of the beam, (ii) a frame number or time step, (iii) a bounding box and / or pixel mask of the detected melt pool area, and (iv) a list of tracker identifiers in the case of intersection events. In some implementations, generating the queue of match scores includes (i) a position P of each event, E , (ii) the last known location P of each beam tracker T , (iii) the number of time steps since the last detection in each beam tracker, Δ T , (iv) the predicted next location P of each beam tracker P , (v) average global beam velocity μ per time step v , and (vi) a user-defined maximum speed V max In some implementations, generating a queue of match scores includes calculating a match score between each event and each beam tracker based on P E and P T The distance between max * Δ T In some implementations, the match score corresponds to the magnitude of acceleration / deceleration normalized to the average speed, hereinafter:

number

number

number

[0075] The method also includes processing (610) the queue of match scores to identify a respective beam tracker of the plurality of beam trackers 444 to pair with each event of the one or more events. In some implementations, processing the queue of match scores includes determining whether each event of the one or more events can be paired with a respective beam tracker of the plurality of beam trackers based on the queue of match scores. An event is identified if its corresponding match score in the queue of match scores exceeds a predetermined threshold (e.g., P E and P T The distance between max * Δ TA beam tracker may be paired with the beam tracker only if its match score is less than 1. The method also includes, for each beam tracker of the plurality of beam trackers, removing the respective beam tracker from the queue when the respective beam tracker cannot be paired with any event of the one or more events. When the respective beam tracker may be paired with two or more events of the one or more events, the method sorts the pairs and selects the best-matching event based on the match score of the pair. When the respective beam tracker may be paired with only one event of the one or more events, and only one event may be paired with two or more beam trackers, the method also includes adding the paired beam tracker to an intersection list of only one event and updating the intersection time step of the paired beam tracker. When (i) each beam tracker corresponds to a best match of only one event, or (ii) only one event can be paired with only the respective beam tracker, the system includes only the one event in the list of detected event objects for each beam tracker, removes from the queue all pairs that include either only the one event or the respective beam tracker, and removes the respective beam tracker from the queue.

[0076] The method also includes processing (612) the unpaired events of the one or more events to associate the unpaired events with (i) a corresponding tracker of the plurality of beam trackers, or (ii) a new tracker, including adding the new tracker to the plurality of beam trackers. In some implementations, processing the unpaired events includes calculating a crossing time threshold CTT, which is the maximum amount of time a beam takes to separate or re-emerge after crossing another beam, by the formula

number

number

[0077] In some implementations, the method further includes extracting a molten pool image for each beam tracker of the plurality of beam trackers that are active in each video frame of the video feed.

[0078] The terms used in describing the present invention herein are for the purpose of describing particular implementations only and are not intended to be limiting of the present invention. As used in the description of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context indicates otherwise. The term "and / or," as used herein, will also be understood to refer to and include any possible combination of one or more of the associated listed items. It will be further understood that the terms "comprises" and / or "comprising," as used herein, specify the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0079] The foregoing description has been set forth with reference to specific implementations for purposes of explanation. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The implementations have been chosen and described to best explain the principles of the invention and its practical application, and thereby enable those skilled in the art to best utilize the invention and various implementations with various modifications suitable for the particular uses contemplated. [Explanation of symbols]

[0080] 100 processes 200 ways 202 Input beam tracker 300 beam path 302 Color Mapping 400 computing devices 402 processing units / cores, GPU / CPU 404 Network or other communications interface 406 User Interface 408 Display Devices 410 Input Devices or Mechanisms 412 communication bus 414 memory 416 Operating Systems 418 Communication Module 420 data visualization applications or modules 422 Imaging Module 424 images 426 Gradient Mask Generation Module 428 Gradient Mask 430 pixel mask generation module 432 pixel mask 434 Beam Location Module 436 Beam Location 438 Beam Tracking Module 440 Events 442 Match Score 444 Beam Tracker 446 Detection Event Object 500 ways 600 ways

Claims

1. 1. A method for tracking multiple beams in a weld pool, comprising: acquiring an input video feed including a plurality of frames having a plurality of beams for a weld pool; creating, for each frame of the plurality of frames of the input video feed, a gradient mask that identifies one or more pixels in the respective frame that relate to a molten pool; detecting and filtering contiguous regions based on the gradient mask to obtain a pixel mask for each contour of the molten pool; identifying the location of one or more beams within the weld pool based on the pixel mask of each contour; A method comprising:

2. identifying the one or more pixels For each pixel in the respective frame, calculating a time-based derivative of the intensity of the respective pixel based on the respective frame and a previous frame of the input video feed; determining potential locations of the weld pool based on the time-based derivative of the intensity of each pixel; and 2. The method of claim 1, comprising:

3. determining potential locations of the molten pool; 3. The method of claim 2, comprising calculating at least one of: (i) a relative intensity of each pixel to a background image; (ii) an emission intensity of a material used for welding in the near-infrared (NIR) spectrum and the infrared (IR) spectrum; or (iii) a drop in intensity.

4. determining potential locations of the molten pool; 3. The method of claim 2, comprising thresholding the time-based derivative of each pixel with a high-low pass filter to create the gradient mask, thereby marking pixels that either experience a sudden increase or decrease in value as potential locations of a weld pool.

5. 5. The method of claim 4, wherein the high-low pass filter is applied to the time-based derivative and includes two thresholds including: (i) a low value that is a negative number, where a delta value below the low value represents a phase change from powder to liquid metal; and (ii) a high value that is a positive number, above which either heat is increasing or a beam is crossing on a pixel.

6. The method of claim 4 further comprising defining a respective threshold value for each weld type.

7. The method of claim 4 , further comprising using a morphological closing filter on the gradient mask to fill in any gaps between isolated pixels surrounding potential weld pool locations.

8. obtaining the pixel mask for each contour of the molten pool; detecting contours based on the gradient mask; filtering the contour by size with a high pass filter to remove any regions that are too small to plausibly be a weld pool; outputting said pixel mask for each remaining contour; 2. The method of claim 1, comprising:

9. identifying the location of the one or more beams within each weld pool; identifying peak intensities by calculating local maxima within the pixel mask for each contour; identifying a missing beam for each contour by determining whether the peak intensity is below a predetermined range; applying a high pass threshold to a surrounding area based on the peak intensity to determine a spot area; calculating coordinates of the location of the one or more beams based on an average value or a centroid of the spot area; 2. The method of claim 1, comprising:

10. identifying the location of the one or more beams within each weld pool; 10. The method of claim 9, further comprising: (i) measuring an overall size and aspect ratio of the weld pool; and (ii) responsive to determining that the weld pool has a significantly increased size or a moderately increased size and an unusual aspect ratio, locating multiple beams within the weld pool by detecting a bimodal peak within a high intensity range.

11. 1. A method for tracking multiple beams in a weld pool, comprising: acquiring an input video feed including a plurality of frames having a plurality of beams for a weld pool; For each frame of the input video feed: detecting one or more events, each event corresponding to a location of a respective beam within the weld pool in a respective frame; generating a queue of match scores between the one or more events and a plurality of beam trackers, each match score representing a degree of match between a respective event of the one or more events and a respective beam tracker of the plurality of beam trackers, each beam tracker being associated with a list of detection event objects, each detection event object being associated with a beam of the one or more beams; processing the queue of match scores to identify a respective beam tracker of the plurality of beam trackers to pair with each event of the one or more events; processing unpaired events of the one or more events to associate the unpaired events with (i) a corresponding tracker of the plurality of beam trackers, or (ii) a new tracker, including adding the new tracker to the plurality of beam trackers; A method comprising:

12. detecting the one or more events includes: identifying a molten pool by detecting and filtering a contiguous region within each of the frames; storing the bounding box of the weld pool in a new detection event object for an event of the one or more events; determining a location of the beam within the weld pool based on detecting a peak intensity and high-pass thresholding a surrounding area at the peak intensity; storing the beam location coordinates in the new detection event object; 12. The method of claim 11, comprising:

13. detecting the one or more events includes: determining whether the weld pool contains multiple beam locations by measuring the overall size and aspect ratio of the weld pool; generating a second detection event object for a location of a second beam within the weld pool in response to determining that the weld pool includes multiple beams and in response to determining that the peak intensity has a bimodal peak; 13. The method of claim 12, further comprising:

14. generating said queue of match scores comprises: (i) the position P of each of the events E (ii) the last known location P of each of said beam trackers T (iii) the number of time steps since the last detection in each of the beam trackers, Δ T (iv) the predicted next location P of each beam tracker P , (v) average global beam velocity μ per time step v , and (vi) a user-defined maximum speed V max 12. The method of claim 11, further comprising calculating a match score between the respective event and the respective beam tracker based on:

15. generating said queue of match scores comprises: P E and P T The distance between max * Δ T 15. The method of claim 14, further comprising the step of forgoing adding the match score to the queue in response to a determination that the match score exceeds

16. The match score is: i. Corresponding to the magnitude of acceleration / deceleration normalized to the average velocity, [Equation 1] ii. The angle between the vector to the event and the vector to the prediction, normalized to 180, is: [Equation 2] iii. corresponds to the mean deviation from the mean speed, normalized to the mean speed, plus the penalty for exceeding the mean speed, [Equation 3] and iv. The method of claim 14, wherein the event index is an ordered structure comprising a vector / list of indices of the respective events and the respective beam trackers.

17. processing the queue of match scores, determining whether each event of the one or more events can be paired with a respective beam tracker of the plurality of beam trackers based on the queue of match scores, wherein an event can be paired with a beam tracker only if the corresponding match score of the event in the queue of match scores exceeds a predetermined threshold; For each beam tracker of the plurality of beam trackers: removing the respective beam tracker from the queue in response to determining that the respective beam tracker cannot be paired with any of the one or more events; responsive to determining that the respective beam tracker may be paired with two or more of the one or more events, sorting the pairs and selecting a best matching event based on a match score of the pair; in response to determining that the respective beam tracker may be paired with only one event of the one or more events; in response to determining that the single event may be paired with two or more beam trackers; adding the paired beam trackers to an intersection list of the single event and updating the intersection time step of the paired beam trackers; (i) the respective beam tracker corresponds to a best match for the single event, or (ii) the single event can be paired with only the respective beam tracker; including the single event in the list of detected event objects for the respective beam tracker; removing from the queue all pairs that include either the single event or the respective beam tracker; removing the respective beam tracker from the queue; 12. The method of claim 11, comprising:

18. processing the unpaired events, The crossing time threshold (CTT), which is the maximum amount of time a beam can take to separate or reappear after crossing another beam, is defined as [Equation 4] Calculating based on μ v is the average global beam velocity per time step, and identifying one or more beam trackers from the plurality of beam trackers to pair with each unpaired event based on checking all events for all beam trackers with timestamps within the last CTT time step for entries in the intersection list; 12. The method of claim 11, comprising:

19. processing the unpaired events, While checking all events for all beam trackers with timestamps within the previous CTT time step for entries in the intersection list, (a) On machines having a pulsed beam or a beam that cycles rapidly between locations, [Equation 5] and (b) Otherwise, to account for crossover events, IT = CTT 20. The method of claim 18, further comprising the step of ignoring beams that are not detected within a time period IT calculated based on a threshold that is the largest of:

20. The method of claim 11 , further comprising extracting a molten pool image for each beam tracker of the plurality of beam trackers that is active in each video frame of the input video feed.

21. 12. The method of claim 11, wherein each beam tracker further includes: (i) a unique identifier; (ii) a video output file or object for the video being extracted; (iii) a cache list or vector of images; and (iv) an integer representing the last intersection time step.

22. The method of claim 11 , wherein each detection event object includes: (i) the X / Y coordinates of the beam, (ii) the frame number or time step, (iii) the bounding box and / or pixel mask of the detected weld pool area, and (iv) a list of tracker identifiers in case of intersection events.

23. one or more processors; Memory and Including, 23. A computer system, wherein the memory stores one or more programs configured for execution by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 1 to 22.

24. 23. A non-transitory computer-readable storage medium storing one or more programs configured for execution by a computer system having one or more processors and a memory, the one or more programs including instructions for performing the method of any one of claims 1 to 22.

Images